Exadata X5-2 Full Rack
 
Exadata X5-8 Hardware
 
Exadata X6-2 Hardware
 
Exadata X6-8 Hardware
 
Exadata X5-2 Hardware
 
Exadata X4-8 Hardware
 
Oracle SuperCluster T5-8 Hardware
 
Oracle SuperCluster M6-32 Hardware
 

PLA-Support>Sun Systems>x86>Engineered Systems HW>SN-x64: EXADATA
 

Exadata X5/X6 reports "Disk controller was hung. Cell was power cycled to stop the hang." on SW images prior to 12.1.2.3.3.161109

In this Document

Symptoms

Cause

Solution

References

Applies to:

Symptoms

Exadata X5/X6 reports "Disk controller was hung. Cell was power cycled to stop the hang." on SW images prior to 12.1.2.3.3.161109

Note: Exadata X5-2L/X6-2L Extreme Flash Storage Servers do not have a SAS HBA and are not affected.
Note: Exadata systems older than X5-2 use a different SAS HBA and are not affected.
Note: Exadata X4-8 systems with X5-2L Storage Servers only applies to the Storage Servers; the X4-8 DB nodes use a different SAS HBA and is not affected.

Issue: The server will alert with this (cellcli or dbmcli - depending on server type that saw the event):

In some cases the server's logging a reset event may be due to HBA controller Correctable Error that is repeated thousands of times in the SAS HBA firmware terminal log (fwtermlog) such as:

In this case the error is correctable and really only occurred once, but a bug in the error handling routine causes it to be repeated thousands of times before it forcibly resets.

In some cases the reset event may not have anything useful logged as to the reason in the HBA controller logs, reporting that the firmware was hung:

IMPORTANT NOTE: It is possible due to another image issue in 12.1.2.2.2 and earlier that the SAS HBA firmware logs do not contain any information due to a incorrect setting that clears the logs on the power cycle. If this is the case, please ensure also the procedure in Note 2135119.1 is completed to ensure persistent logging is enabled in addition to this solution. The additional steps are included below.

Workaround: None, the power cycle recovers the HBA back to an operational state, as designed by Exadata server monitoring service (MS).

Cause

Unpublished HBA Firmware bug 21669752 causes the controller to hang while correcting an error.

Unpublished HBA Firmware bugs 23086151 and 23625327 cause the controller to hang due to a firmware hang issue.

Solution

The solution is to update the SAS HBA firmware to version 24.3.0-0084 (or later).
If a controller hang or reset event has occurred with other messages logged, or the event occurs on a system that already has SAS HBA firmware 24.3.0-0084, then a SR should be opened and a Sundiag and diagpack provided for the event uploaded for analysis and action plan.

1. Check the image version of the server and firmware version of the SAS HBA: 

If the server is running SW Image 12.1.2.3.3.161109 or later, then the problem does not apply.

These images have the firmware fix in "FW Package Build: 24.3.0-0084".

If a controller hang or reset event has occurred with other messages logged, or the event occurs on a system that already has SAS HBA firmware 24.3.0-0084, then a SR should be opened and a Sundiag and diagpack provided for the event uploaded for analysis and action plan.

2. Update the SW image to 12.1.2.3.3.161109 or later, which contains the firmware fix 24.3.0-0084 per the example above. For how to update image, refer to MOS Note 888828.1. 

The first preference for resolving this issue is to update image.  If the server is not able to be updated with a later image at this time, then the SAS HBA firmware only may be temporarily updated to address this issue on systems that have had a failure as follows, using the firmware package "MR_6.3.8.4_24.3.0-0084.rom" attached to this Note as follows:

a) Download the firmware package "MR_6.3.8.4_24.3.0-0084.rom" attached to this Note, and copy it to the /tmp directory on the server to be updated. 

The md5sum of the file is:

b) Prepare the server for maintenance as follows:

Exadata Storage Servers (based on X5-2L and X6-2L)

i. ASM drops a disk shortly after it/they are taken offline. The default DISK_REPAIR_TIME
attribute value of 3.6hrs should be adequate for replacing components, but may have been
changed by the Customer. To check this parameter, have the Customer log into ASM and
perform the following query:

As long as the value is large enough to comfortably perform the upgrade in a
storage cell, there is no need to change it.

ii. Check if ASM will be OK if the grid disks go OFFLINE.

If one or more disks return asmdeactivationoutcome='No', you should wait for some time
and repeat the query until all disks return asmdeactivationoutcome='Yes'.

iii. Run cellcli command to Inactivate all grid disks on the cell you wish to power down/reboot.
(this could take up to 10 minutes or longer)

iv. Execute the command below and the output should show asmmodestatus='UNUSED' or
'OFFLINE' and ‘asmdeactivationoutcome=Yes’ for all griddisks once the disks are offline and
inactive in ASM.

 v) Disable Exadata Storage Server services with the following command as 'root' user:

Exadata DB Nodes (based on X5-2, X6-2 and X5-8)

Linux DB Nodes:

i) Shutdown and disable auto-start of CRS services:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.

In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

ii)  Validate CRS is down cleanly. There should be no processes running.

iii)  Disable Exadata DB Node management services with the following command as 'root' user:

OVM DB Nodes:

 i) See what user domains are running (record result)

Connect to the management domain (domain zero, or dom0).  This is an example with just two domains and the management domain Domain-0

ii) Connect to each domain using the command:

where domainname would be dm01db01vm01 or dm01db01vm02 if using the above examples.

iii) Shut down any instances of CRS on that domain:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.

In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

iv) Validate CRS is down cleanly. There should be no processes running.

v) Press CTRL+] to disconnect from the console.

vi) Repeat steps ii - v on each running domain.

vii) Shutdown all user domains from dom0

viii) Repeat step 1 to see what user domains are running. It should be only Domain-0.

ix) Disable user domains from auto starting during dom0 boot after firmware has been updated.

 x) Disable Exadata DB Node management services with the following command as 'root' user:

c) Upgrade the server's SAS HBA firmware with the following command as 'root' user:

Upon completion of the firmware upgrade, the server will automatically reboot. There may be periods of time during the update where the output to the screen stops, which is expected - please be patient. This takes ~10 minutes to get to the reboot, and ~15 minutes to complete the entire process including rebooting the cell, excluding disk re-synchronization or CRS start time. The time may be longer on X5-8 DB nodes.  There may be 2 reboots during the process.

d) Verify the server's SAS HBA firmware is updated:

The firmware package with the bug fix is 24.3.0-0084. 

3. Verify the server's disks after firmware update and bring back online its services as follows:

Exadata Storage Servers (based on X5-2L and X6-2L):

i. Verify the 12 disks are visible. The following command should show 12 disks:

ii. Activate the grid disks.

iii. Verify all grid disks show 'active':

iv. Verify all grid disks have been successfully put online using the following command. Wait until asmmodestatus is ONLINE for all grid disks. The following is an example of the output early in the activation process.

Notice in the above example that RECO_CD_00_dmorlx8cel01 is still in the 'SYNCING' process. Oracle ASM synchronization is only complete when ALL grid disks show ‘asmmodestatus=ONLINE’. This process can take some time depending on how busy the machine is, and has been while this individual server was down for repair.

Exadata DB Nodes (based on X5-2, X6-2 and X5-8)

Linux DB Nodes:

i) Verify all the disks are visible to the system and in 'normal' status.

 There should be 4 or 8 disks depending on the DB node model.

ii) Startup CRS and re-enable autostart of crs. After the OS is up, the Customer DBA should validate that CRS is running. As root execute:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.

In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.
Example output when all is online is: 

iii) Validate that instances are running:

It should return a record for the ASM instance and a record for each database.

OVM DB Nodes:

i) Verify all the disks are visible to the system and in 'normal' status.

There should be 4 or 8 disks depending on the DB node model. 

ii) Re-enable user domains to autostart during Domain-0 boot:

iii) Startup all user domains that are marked for auto start:

iv)  See what user domains are running (compare against result from previously collected data):

 v) if any did not auto-start then Startup a single user domain:

vi)  Check that CRS has started in user domains:

a) Connect to each domain using the command:

where domainname would be dm01db01vm01 or dm01db01vm02 if using the above examples.

b) Any instances of CRS on that domain should have automatically started:

c) Validate that instances are running:

It should return a record for the ASM instance and a record for each database.

d) Press CTRL+] to disconnect from the console.

vii) Repeat step (vi) on each running domain.

4. Repeat the above steps to update the firmware on each storage server and DB node, as needed.

If the image is version 12.1.2.3.0 or later, then the procedure is complete.  The following additional steps are required for images 12.1.2.2.2 or below (as taken from Note 2135119.1):

5. To verify the current battery status for the fwtermlog setting, on each server, as 'root' login, execute:  

We should see that the battery mode is off for the fwtermlog.

6. Turn on use of the battery for maintaining the fwtermlog across server reboots and power cycles:

Running the above command on the server will not have any impact on running services.

This change is persistent across server reboots or power cycles and is only unset by command.

References

This solution has no attachment