Oracle ZFS Storage Appliance: Node not booting. No display via serial connection to ILOM

Asset ID:	1-72-2390835.1
Update Date:	2018-04-30
Keywords:

Solution Type Problem Resolution Sure

Solution 2390835.1 : Oracle ZFS Storage Appliance: Node not booting. No display via serial connection to ILOM

Applies to:

Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Node not booting

Unable to access ILOM

Amber led blinking on both power supplies.

No display when connected direct monitor to server. All disk trays were receiving power.

Changes

System was not in use for a while. No known changes.

Cause

Multiple hardware failures.

Solution

1. Verify and confirm with customer that external Power Supply is working fine.

2. Power cycle the node and try to power it on:

(2.1) Power off the node and pull out all Power Supply cables on the node from rear end
(2.2) Press and hold the power button on the node for 10seconds.
(2.3) Connect power cables to the node. If possible test with any other Power Supply which is working fine.
(2.4) Power on the node.
(2.5) Collect full ILOM snapshot if ILOM is alive

3. If the above steps fail to bring up the node and also no output via ILOM serial console, System Board will need replacement. Replace faulty System Board.

At this stage we should have access to ILOM, even if the Node does not boot up. We will have to look into the ILOM snapshot for more information.

4. If ILOM is 2.x , please run these IPMI tool commands from a Linux machine within customer network and capture the output:

ipmitool -H <ipaddress of SP or CMM> -U root fru
ipmitool -H <ipaddress of SP or CMM> -U root sel elist
ipmitool -H <ipaddress of SP or CMM> -U root -v sdr
ipmitool -H <ipaddress of SP or CMM> -U root sdr elist
ipmitool -H <ipaddress of SP or CMM> -U root sdr list
ipmitool -H <ipaddress of SP or CMM> -U root chassis status
ipmitool -H <ipaddress of SP or CMM> -U root sunoem led get
ipmitool -H <ipaddress of SP or CMM> -U root sensor
ipmitool -H <ipaddress of SP or CMM> -U root mc info

5. Run these CLI commands via SSH or Serial Connection to ILOM and capture the output:

show -l all
show -o table -level all
show /SP/logs/event/list
show faulty

6. If you are still unable to identify the failures from ILOM snapshot, IPMI tool commands or other CLI commands, then we need to bring the server to minimal config:

Note: ILOM running on 2.x may not show correct IPMI sensor data. In this case , Upgrade the ILOM to latest for the testing and then bring the server to minimal config

(6.1) Remove all PCIE cards and risers
(6.2) Disconnect the connector board from the PDB and disconnect all front panel cables
(6.3) Remove all DIMMs
(6.4) Remove PS1 and power on with single PSU0
(6.5) Check ILOM and see the system board components are visible and sensor data readings
(6.6) Swap the PSU and check
(6.7) If with this config, ILOM showing installed components (CPU,BIOS and sensor values for CPU and other voltage sensors)
(6.8) Then add components in the following order. After adding each component in above steps, need to verify the ILOM sensor and component status, in any step if the ILOM not showing up properly, remove the component and kept aside,continue with next part

- PSU1
- Connect Connector board only (do not connect fan board and DBP)
- Connect Fan board0
- Connect Fan board1
- Connect Disk back plane
- Connect DVD
- Connect disks one by one
- Connect PCI Risers
- Connect PCI card
- Install DIMMs pair by pair - D6 and D7, then D5 and D4.. etc for both CPUs

(6.9) Once the server is fully configured, power on the server and check

7. At step 6.7 If you are still seeing the same issue, i.e no reading for ILOM sensors and no components reported for CPU and BIOS, Then replace the System Board again and start from step 6.6

8. If Node is powered on and you are unable to access it, make sure BIOS is set to appropriate values after System Board replacement.

9. Perform health check.

References

<NOTE:1635390.1> - Sun Storage 7000 Unified Storage System: 7310 Array Will Not Boot
<NOTE:2107944.1> - Sun Storage 7000 Unified Storage System: 7410 - Required BIOS Settings

Attachments

This solution has no attachment