Oracle SuperCluster T5-8 Full Rack
 
Exadata X3-2 Hardware
 
SPARC SuperCluster T4-4 Full Rack
 
Exadata X4-2 Hardware
 
Exadata X4-2 Quarter Rack
 
Oracle SuperCluster T5-8 Half Rack
 
Exadata X3-2 Half Rack
 
Exadata X4-8 Hardware
 
Exadata X3-2 Full Rack
 
Exadata X4-2 Half Rack
 
Zero Data Loss Recovery Appliance X4 Hardware
 
Exadata X3-8 Hardware
 
Exadata X4-2 Full Rack
 
Exadata X3-2 Eighth Rack
 
SPARC SuperCluster T4-4 Half Rack
 
Oracle SuperCluster T5-8 Hardware
 
SPARC SuperCluster T4-4
 
Exadata X3-2 Quarter Rack
 
Exadata X4-2 Eighth Rack
 
Exadata X3-8b Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

How-to procedure for replacing a HBA Battery Backup Unit (BBU) on an Exadata Database Machine node (X3-2, X3-8, X4-2, X4-8).

Applies to:

Goal

 How to procedure for replacing a HBA Battery Backup Unit (BBU) on an Exadata Database Machine node (X3-2, X3-8, X4-2 and X4-8).

Solution

DISPATCH INSTRUCTIONS WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?: Exadata trained
TIME ESTIMATE: 60 minutes
TASK COMPLEXITY: 2

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW:

The Battery Backup Unit (BBU) on the RAID HBA needs replacement in Exadata Database Machine X3-2, X3-8, X4-2 or X4-8.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

The instructions below assume the customer DBA is available and working with the field engineer onsite to manage the host OS and DB/ASM services. They are provided here to allow the FE to have all the available steps needed when onsite, and can be done by the FE if the customer DBA wants or allows or needs help with their steps.

On certain X3-2/X4-2/X4-8 DB Nodes, X3-2/X4-2 and X3-8/X4-8 Storage Servers, the BBU is remote mounted and may not require a system shutdown to access, however it must be prepared for removal from the RAID HBA before it can be removed and replaced in order to avoid the risk of data corruption to the disk volumes. In certain revisions of firmware, the server operating system does need to be shutdown in order to prevent an unplanned outage possibility.

Note: there is no remote mount BBU option for X3-8 DB Nodes, and these always need to be shutdown for  replacement. 

1. Identify the image version currently running on the server in the rack that requires service.

For Exadata Storage Servers, as 'celladmin' or 'root' user:

For Database Nodes, as 'root' user:

The version information is 5 digits e.g. 11.2.3.2.1, the latter part is the image version date. 

2. Revert all the RAID disk volumes to WriteThrough mode to ensure all data in the RAID cache memory is flushed to disk and not lost when replacement of the Battery occurs:

For Exadata DB Nodes (except X3-8) with image 12.1.2.1.0 and later:

i. Drop the BBU to prepare it for replacement:

ii. Verify the BBU is dropped from use:

iii. Verify the current cache policy for all logical volumes is now WriteThrough which does not use the battery:

This may take several minutes to completely flush the cache to disk and disable the battery. Do NOT proceed until the MegaCli64 returns the correct "Current Cache Policy" value.

For Exadata DB Nodes (except X3-8) with image 11.2.3.3.0 through 12.1.1.1.2:

i. Drop the BBU to prepare it for replacement:

ii. Verify the BBU is dropped from use:

iii. Verify the current cache policy for all logical volumes is now WriteThrough which does not use the battery:

This may take several minutes to completely flush the cache to disk and disable the battery. Do NOT proceed until the MegaCli64 returns the correct "Current Cache Policy" value.

For Exadata Storage Servers with image 11.2.3.3.0 and later:

i. Drop the BBU to prepare it for replacement:

ii. Verify the BBU is dropped from use:

iii. Verify the current cache policy for all logical volumes is now WriteThrough which does not use the battery:

This may take several minutes to completely flush the cache to disk and disable the battery. Do NOT proceed until the MegaCli64 returns the correct "Current Cache Policy" value.

For Exadata DB Nodes and Storage Servers with image 11.2.3.2.1 and earlier, and all Exadata X3-8 DB nodes regardless of image version:

i. Set all logical volumes cache policy to WriteThrough cache mode:

ii. Verify the current cache policy for all logical volumes is now WriteThrough which does not use the battery:

This may take several minutes to completely flush the cache to disk and disable the battery. Do NOT proceed until the MegaCli64 returns the correct "Current Cache Policy" value.

3. If the image version is 12.1.2.2.0 or later, and this is not a X3-8 DB node, then no additional preparation is needed and the battery replacement can occur while the system continues to operate - in this case, move to the next action for physical replacement. If the image version is 12.1.2.1.x or earlier, or the replacement is on a X3-8 DB node, then the server operating system needs to be shutdown prior to physical replacement in order to prevent the possibility of an unplanned outage of applications:

For Exadata Storage Servers:

i. For Extended information on this section check MOS Note ID 1188080.1 Steps to shut down or reboot an Exadata storage cell without affecting ASM

This is also documented in the Exadata Database Machine Maintenance Guide chapter 1 section titled "Powering On and Off Oracle Exadata Rack" available on the customer's cell server image in the /opt/oracle/cell/doc directory or online at https://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/general.htm#DBMMN115.

In the following examples the SQL commands should be run by the Customers DBA prior to doing the hardware replacement. These should be done by the field engineer only if the customer directs them to, or is unable to do them. The cellcli commands will need to be run as root.

Note the following when powering off Exadata Storage Servers:

• Verify there are no other storage servers with disk faults. Shutting down a storage server while another disk is fails may result in the running database processes and Oracle ASM to crash if it loses both disks in the partner pair when this server’s disks go offline.
  
  
• Powering off one Exadata Storage Server with no disk faults in the rest of the rack will not affect running database processes or Oracle ASM.
  
  
• All database and Oracle Clusterware processes should be shut down prior to shutting down more than one Exadata Storage Server. Refer to the Exadata Owner’s Guide for details if this is necessary.

ii. ASM drops a disk shortly after it/they are taken offline. Powering off or restarting Exadata Storage Servers can impact database performance if the storage server is offline for longer than the ASM disk repair timer to be restored. The default DISK_REPAIR_TIME attribute value of 3.6hrs should be adequate for replacing components, but may have been changed by the Customer. To check this parameter, have the Customer log into ASM and perform the following query:

As long as the value is large enough to comfortably replace the components being replaced, then there is no need to change it.

iii. Check if ASM will be OK if the grid disks go OFFLINE.

If one or more disks return asmdeactivationoutcome='No', then wait for some time and repeat step #2. Once all disks return asmdeactivationoutcome='Yes', proceed to the next step.

iv. Run cellcli command to Inactivate all grid disks on the cell that needs to be powered down for maintenance. (this could take up to 10 minutes or longer)

v. Execute the command below and the output should show asmmodestatus='UNUSED' or 'OFFLINE' and asmdeactivationoutcome=Yes for all griddisks once the disks are offline and inactive in ASM.

vi. Once all disks are offline and inactive, the customer may shutdown the Cell using the following command:

 When powering off Exadata Storage Servers, all storage services are automatically stopped.

For Exadata DB Nodes:

i. For Extended information on this section check MOS Note ID 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.

For a documentation reference, in the Exadata Database Machine Maintenance Guide, use the section of chapter 1 titled "Powering On and Off Oracle Exadata Rack" available on the customer's cell server image in the /opt/oracle/cell/doc directory or online at https://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/general.htm#DBMMN115.

In the following examples the SQL commands should be run by the Customers DBA prior to doing the hardware replacement. These should be done by the field engineer only if the customer directs them to, or is unable to do them.

ii. Customer should shutdown CRS services prior to powering down the DB node.  As root user do the following:

where GI_HOME environment variable is typically set to "/u01/app/11.2.0/grid" but will depend on the customer's environment.

In the above output the "1" of "+ASM1" refers to the DB node number.
For example, Db node #3 the value would be +ASM3.

iii. Validate CRS is down cleanly. There should be no processes running.

 iv. Shutdown the server operating system:

Linux:

Solaris:

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

Exadata X3-2/X4-2/X4-8 Compute Nodes and X3-2/X3-8/X4-2/X4-8 Storage Cell nodes with the Remote Battery:

These steps are relevant to Exadata nodes based on X3-2/X4-2/X4-8 and X3-2L/X4-2L servers with the remote battery assembly (part 7057184) installed.

1.Locate the battery slot marked with an orange and white BBU label.

X3-2/X4-2 Compute nodes this is the upper right-most slot on the front of the chassis labeled BBU (previously designated "HDD7")

X4-8 Compute nodes this is in the lower slot second from the left on the rear of the chassis labeled BBU (previously designated “HDD4”)

X3-2L/X4-2L Storage cells this is the right-hand slot on the rear of the chassis above PS1, labeled BBU (previously designated "REAR HDD 1")

2.Unlatch and carefully slide out the old BBU carrier.

3.Insert and carefully slide in the new BBU carrier, and latch it closed

Exadata X3-2L/X4-2L Storage Cell nodes without the Remote Battery:

Replace the existing HBA BBU with a remote-mounted battery kit (part 7060020) following the CAP detailed in MOS Note 1561949.1. This includes a remote battery assembly (part 7057184).

Exadata X3-2/X4-2 Database Machine Compute nodes without the Remote Battery:

Replace the existing HBA BBU with a remote-mounted battery kit (part 7060020) following the CAP detailed in MOS Note 1561949.1. This includes a remote battery assembly (part 7057184).

Exadata X3-8 Database Machine Compute nodes:

These steps are relevant to Exadata nodes based on X2-8 servers (formerly x4800m2) with direct attached battery (part 371-4982 or 7050794) .

1.Remove CMOD0 from the server setting it on a flat, antistatic surface.

2.Remove the CMOD top cover.

3.Remove the HBA REM with BBU attached:

a.Lift the REM ejector handle and rotate it to its fully open position

b.Lift the connector end of the REM and pull the REM away from the retaining clip on the front support bracket.

4.Remove the old BBU from the REM:

a.Use a No. 1 Phillips screwdriver to remove the 3 retaining screws that secure the battery to the REM card. Do NOT attempt to remove any screws from the top side of the REM and battery pack – those screws hold the standoffs that provide the bottom screw holes and should remain with the battery pack.

b.Detach the battery pack including circuit board from the REM by gently lifting it from its circuit board connector.

5.Install the new BBU on the REM:

a.Attach the battery pack circuit board connector to mate with the REM’s connector.

b.Use a No. 1 Phillips screwdriver to to secure the battery to the REM. If the BBU comes with a package of new screws, then use those new screws - do not re-use the screws from the old BBU attachment.

6.Re-install the HBA REM with BBU attached:

a.Ensure that the REM ejector lever is in the closed position. The lever should be flat with the REM support bracket.

b.Position the REM so that the battery is facing downward and the connector is aligned with the connector on the motherboard.

c.Slip the opposite end of the REM under the retaining clips on the front support bracket
and ensure that the notch on the edge of the REM is positioned around the alignment
post on the bracket.

d.Carefully lower and position the connector end of the REM until the REM contacts the connector on the motherboard, ensuring that the connectors are aligned. To seat the connector, carefully push the REM downward until it is in a level position.

7.Install the cover on the CMOD.

8.Return the CMOD back into the unit in CMOD0 slot.

OBTAIN CUSTOMER ACCEPTANCE WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

These steps are applicable to all systems after completing the physical replacement above. These should be done in co-operation with the customer’s administrator to complete the procedure and verification prior to the field engineer leaving the customer site.

The following additional steps should be completed to return the system to service:

1. For systems (other than X3-8 DB Nodes) with image 12.1.2.2.0 or later, skip to step 2. For systems with image 12.1.2.1.2 and earlier, and for X3-8 DB nodes, that required the system to be shutdown for replacement of the battery or installation of the remote battery kit, startup the system as follows:

a)  Connect to the server’s console.  To connect to the console through ILOM:

From the ILOM Web browser (preferred):
Access the “Remote Control → Redirection” tab and then click on the “Launch Remote Console” button. (On ILOM 3.1.x systems, the console button can be launched from the initial Summary Information screen).

From the ILOM CLI:

b) Power on the server using the front power button or through ILOM

c) Monitor the system booting on the server's console. Watch in particular, the LSI controller BIOS while it is loading. If it gives a warning message regarding drives with preserved cache, then choose “D” to discard the cache and continue. This is not an issue as the disk will get re-synced after boot by ASM. If it gives a warning message regarding drives are in write-through mode due to a low battery, then chose to continue.

The Exadata boot should continue normally after, showing the Exadata boot splash screen and continue with normal OS boot messages. Note there may be a long pause between screen outputs on the ILOM serial console during subsequent boot steps as the default console is the graphics, and the Exadata boot splash screen will not display.

2. Verify the RAID HBA controller can see the new battery. Login as ‘root’ user and run the following commands:

a) Verify the new battery is seen as "Present" by the RAID HBA controller. Note, it may take up to 24 hours for the new BBU battery to be detected, although typical time ranges from 1 minute to 1 hour:

Linux:

Solaris:

b) Verify the new battery status is charging. It may take up to 24 hours for the new BBU battery to be detected, although typical time ranges from 1 minute to 1 hour.

Linux:

Solaris:

There will be a page full of output detailing the battery information and status. If there is any type of error reading the battery status, then further investigation by Oracle Support to identify and correct the problem should be requested before continuing.

3. Set all logical drives cache policy to WriteBack cache mode to ensure the RAID volumes are being cached using the new battery:

For Exadata DB Nodes with image 12.1.2.1.0 and later:

i. Re-enable the BBU after replacement:

ii. Verify the BBU is now in use:

For Exadata DB Nodes with image 11.2.3.3.0 through 12.1.1.1.2:

i. Re-enable the BBU after replacement:

ii. Verify the BBU is now in use:

For Exadata Storage Servers with image 11.2.3.3.0 and later:

i. Re-enable the BBU after replacement:

ii. Verify the BBU is now in use:

For Exadata DB Nodes and Storage Servers with image 11.2.3.2.1 and earlier:

i. Set all logical volumes cache policy to WriteBack cache mode:

ii. Verify the current cache policy for all logical volumes is now WriteBack which uses the battery:

If the current cache policy is WriteThrough mode, and not WriteBack, then the battery may not have charged sufficiently to allow the cache to be used. Monitor for 3 to 4 hours and re-check the status. If the current cache policy stays as “WriteThrough” mode for a prolonged period of time and the battery status can be properly read by the HBA as above, then further investigation by Oracle Support to identify and correct the problem should be requested before continuing.

4. For systems (other than X3-8 DB Nodes) with image 12.1.2.2.0 or later, no further action is necessary. For systems with image 12.1.2.1.2 and earlier, and for X3-8 DB nodes, that required the system to be shutdown for replacement of the battery or installation of the remote battery kit, return the services back to normal operation:

For Exadata Storage Servers:

Use the following to return the Cell to service:

a) Activate the grid disks:

b) Issue the command below and all disks should show 'active':    

c) Verify all grid disks have been successfully put online using the following command. Wait until 'asmmodestatus' is in status 'ONLINE' for all grid disks. The following is an example of the output early in the activation process.

Notice in the above example that 'RECO_CD_00_dmorlx8cel01' is still in the 'SYNCING'  process. Oracle ASM synchronization is only complete when ALL grid disks show ‘asmmodestatus=ONLINE’.  This process can take some time depending on how busy the machine is, and has been while this individual server was down for repair.

For Exadata DB Nodes:

The DB services should start automatically. Use the following to verify:

a) Validate that CRS is running, as ‘root’ user execute the following:

b) In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

Validate that instances are running:

It should return a record for ASM instance and a record for each database.

PARTS NOTE:

371-4982 6Gigabit SAS RAID PCI Battery - Module ( LION), BBU-08.  Battery only, use for direct attachment to HBA.

7050794 6Gigabit SAS RAID PCI Battery - Module ( LION), BBU-08, RoHS2013.  Battery only, use for direct attachment to HBA.

7057184 6Gigabit SAS RAID PCI Battery - Remote Mount Assembly (LION, BBU-08). Battery mounted on disk tray sled, use for remote battery replacement, includes battery 7050794.

7060020 Remote Mount Battery Assembly Kit.  Use to upgrade a compatible system from direct attachment to remote battery. The kit includes all parts for 1U and 2U systems, and includes 1x 7057184 battery assembly.

REFERENCE INFORMATION:

LSI MegaRAID User’s Guide - http://www.avagotech.com/support/oem/oracle/6gb/sg_x_sas6-r-int-z

Exadata documentation library for customers - https://docs.oracle.com/cd/E50790_01/doc/index.htm in the "Oracle® Exadata Database Machine Maintenance Guide"

Note ID 1561949.1 How to install or replace the remote mount battery kit in a compatible Engineered System

• Bug 19879870 Exadata X4-2 database node shuts down when replacing external BBU - fixed in image 12.1.2.2.0.
• Bug 22992608 bbu drop for replacement and reenable aren't consistent with bbu charge behavior

This solution has no attachment