How to Replace an Exadata or SuperCluster Storage Server Sun Flash Accelerator F80 PCIe Card

Asset ID:	1-71-1633762.1
Update Date:	2018-04-24
Keywords:

Solution Type Technical Instruction Sure

Solution 1633762.1 : How to Replace an Exadata or SuperCluster Storage Server Sun Flash Accelerator F80 PCIe Card

Applies to:

Oracle SuperCluster M6-32 Hardware - Version All Versions and later
Exadata X4-2 Half Rack - Version All Versions and later
Exadata X3-8b Hardware - Version All Versions and later
Exadata X4-2 Eighth Rack - Version All Versions and later
Exadata X4-2 Quarter Rack - Version All Versions and later
Information in this document applies to any platform.

Goal

Procedure for How to Replace a Sun Flash Accelerator F80 PCIe Card in an Exadata Storage Server (Cell) without loss of data or Exadata service.

Solution

DISPATCH INSTRUCTIONS:
The following information will be required prior to dispatch of a replacement:

Type of Exadata (X4-2) / Exadata Storage Expansion Rack / SPARC SuperCluster
Type of storage cell/Node (X4-2L).
Name/location of storage cell
PCI Slot number of failed card
Image Version (output of "imageinfo -active")

Special Instructions for Dispatch are required for this part.

For Attention of Dispatcher:

The parts required in this action plan may be available as spares owned by the customer, which they received with the Engineered System. (These are sometimes referred to as ride-along spares.)

If parts are not available to meet the customer preferred delivery time/planned end date, then request TAM or field manager to contact the customer, and ask if the customer has parts available, and would be prepared to use them.

If customer spare parts are used, inform the customer that Oracle will replenish the customer part stock as soon as we can. More details on this process can be found in GDMR procedure "Handling Where No Parts Available" step 2: https://ptp.oraclecorp.com/pls/apex/f?p=151:138:38504529393::::DN,BRNID,DP,P138_DLID:2,86687,4,9082,

WHAT SKILLS DOES THE ENGINEER NEED:
The engineer must be Exadata trained, have familiarity with the storage cells and replacing hard drives.

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER INSTRUCTIONS:
PROBLEM OVERVIEW:
There is a failed Sun Flash Accelerator F80 PCIe Card in an Exadata Storage Server (Cell) that needs replacing. It may also require a firmware patch update after replacement.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:
The Storage Cell containing the failed F80 card is required to be powered off prior to card replacement.

If requested by Oracle Support, gather all data files produced by lsi_diag.sh and upload that to the SR prior to powering off the node. This must be gathered PRIOR to replacing the card if requested by Oracle Support.

lsi_diag.sh is a data gathering tool for the F80 card that produces a 'min' tarball file and a 'full' tarball file. When sundiag is run on Exadata, it incorporates the 'min' file into the sundiag tarball, and the 'full' file is placed in /tmp for use if required. Not all SR's will require the 'full' file and it will be removed when the system is powered down. If the 'full' file is no longer available prior to replacing the card, then as root user run '/opt/lsidiag/lsi_diag.sh' which will produce a new 'min' and 'full' file in the current user's directory. lsi_diag Full output is REQUIRED for any F80 card returned for CPAS analysis by LSI

It is expected that the customer's DBA has completed these steps prior to arriving to replace the card. The following commands are provided as guidance in case the customer needs assistance checking the status of the system prior to replacement. If the customer or the FSE requires more assistance prior to the physical replacement of the device, EEST/TSC should be contacted.

Locate the server in the rack being serviced. The cell server within the rack can be determined from the hostname usually, and the known default Exadata server numbering scheme. Exadata Storage Servers are identified by a number 1 through 18, where 1 is the lowest most Storage Server in the rack installed in RU2, counting up to the top of the rack.

Turn on the locate indicator light ‘on’ for easier identification of the server being repaired. If the server number has been identified then the Locate Button on the front panel may be pressed. To turn on remotely, use either of the following methods:
From a login to the CellCli on Exadata Storage Servers:

CellCli> alter cell led on

From a login to the server’s ILOM:

-> set /SYS/LOCATE value=Fast_Blink
Set 'value' to 'Fast_Blink

From a login to the server’s ‘root’ account:

# ipmitool sunoem cli ‘set /SYS/LOCATE value=Fast_Blink’
Connected. Use ^D to exit.
-> set /SYS/LOCATE value=Fast_Blink
Set 'value' to 'Fast_Blink'

-> Session closed
Disconnected
Determine the active image version of the Exadata Storage Server:

# imageinfo -active

This information will be needed to determine if the replacement needs its firmware updated, and should be provided to the Oracle service engineer performing the replacement.
Shutdown the node for which the Flash F80 PCIe card requires replacement.
1. For Extended information on this section check MOS Note:
  ID 1188080.1 Steps to shut down or reboot an Exadata storage cell without affecting ASM
  
  This is also documented in the Exadata Owner's Guide in chapter 7 section titled "Maintaining Exadata Storage Servers" subsection "Shutting Down Exadata Storage Server" available on the customer's cell server image in the /opt/oracle/cell/doc directory or here:
  http://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-storage-servers.htm#DBMMN21129
  
  In the following examples the SQL commands should be run by the Customers DBA prior to doing the hardware replacement. These should be done by the field engineer only if the customer directs them to, or is unable to do them. The cellcli commands will need to be run as root.
  
  Note the following when powering off Exadata Storage Servers:
  - Verify there are no other storage servers with disk faults. Shutting down a storage server while another disk is fails may result in the running database processes and Oracle ASM to crash if it loses both disks in the partner pair when this server’s disks go offline.
  - Powering off one Exadata Storage Server with no disk faults in the rest of the rack will not affect running database processes or Oracle ASM.
  - All database and Oracle Clusterware processes should be shut down prior to shutting down more than one Exadata Storage Server. Refer to the Exadata Owner’s Guide for details if this is necessary.
2. ASM drops a disk shortly after they are taken offline. Powering off or restarting Exadata Storage Servers can impact database performance if the storage server is offline for longer than the ASM disk repair timer to be restored. The default DISK_REPAIR_TIME attribute value of 3.6hrs should be adequate for replacing components, but may have been changed by the Customer. To check this parameter, have the Customer log into ASM and perform the following query:
  
  SQL> select dg.name,a.value from v$asm_attribute a, v$asm_diskgroup dg where a.name = 'disk_repair_time' and a.group_number = dg.group_number;
  
  As long as the value is large enough to comfortably replace the components being replaced, then there is no need to change it.
3. If the flash card disks are being used as griddisks, then please refer to Note 1545103.1 for additional specific instructions before continuing.
4. Check if ASM will be OK if the grid disks go OFFLINE.
  # cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
  ...sample ...
  DATA_CD_10_dbm1cel01 ONLINE Yes
  DATA_CD_11_dbm1cel01 ONLINE Yes
  RECO_CD_00_dbm1cel01 ONLINE Yes
  RECO_CD_01_dbm1cel01 ONLINE Yes
  ...repeated for all griddisks....
  
  If one or more disks return asmdeactivationoutcome='No', then wait for some time and repeat this command. Once all disks return asmdeactivationoutcome='Yes', proceed to the next step.
5. Identify the FlashCacheMode of the cell by running the following:
  # cellcli -e list cell attributes FlashCacheMode
  
  If FlashCacheMode is "WriteThrough" then continue to step f.   If FlashCacheMode is "WriteBack" then continue here.
  
  IMPORTANT - In WriteBack cache mode, the Flash card can be replaced ONLY after the alert with message "Flash disk can be replaced now" has been generated and there is not rebalance running on ASM.
  
  i. Check 'list alerthistory' and you should see the failed flash disk. All the other flash disks on this flash card are called peers and will have PEER FAILURE status alerts entries as well:
  
  Example - Flash card in PCI Slot 5 has a failed flash disk 0 as reported in 'list alerthistory':
  
  # cellcli -e list alerthistory
  ...
  12_2    2015-06-09T02:57:42+09:00       critical        "Flash disk failed. Status              : FAILED Manufacturer        : Sun Model Number        : Flash Accelerator F80 PCIe Card Size                : 186GB Serial Number       : 11000100657 Firmware            : UIO6 Slot Number         : PCI Slot: 5; FDOM: 0 Cell Disk           : FD_12_dbm1cel05 Grid Disk           : Not configured Board Tracer Number : SP33513621 "
  
  Before the complete card can be replaced, an alert with message "Flash disk can be replaced now" has to be generated for each of the PEERs of the failed Flash Disk. This will guarantee that those flashdisk will flush the cache to the griddisks and will not continue caching new data until the card is replaced.
  
  In this example, Flash Disks FD_13, FD_14 and FD_15 reported the message "Flash disk entered peer failure status. and status will be WARNING - PEER FAILURE as seen in 'list alerthistory'
  
  # cellcli -e list alerthistory
  ...
           13_2    2015-06-09T03:07:08+09:00       critical        "Flash disk can be replaced now. Status              : WARNING - PEER FAILURE Manufacturer        : Sun Model Number        : Flash Accelerator F80 PCIe Card Size                : 186GB Serial Number       : 11000100330 Firmware            : UIO6 Slot Number         : PCI Slot: 5; FDOM: 1 Cell Disk           : FD_13_dbm1cel05 Grid Disk           : Not configured Flash Cache         : Present Flash Log           : Present Board Tracer Number : SP33513621 "
  ...
           14_2    2015-06-09T03:07:08+09:00       critical        ""Flash disk can be replaced now. Status              : WARNING - PEER FAILURE Manufacturer        : Sun Model Number        : Flash Accelerator F80 PCIe Card Size                : 186GB Serial Number       : 11000114871 Firmware            : UIO6 Slot Number         : PCI Slot: 5; FDOM: 2 Cell Disk           : FD_14_dbm1cel05 Grid Disk           : Not configured Flash Cache         : Present Flash Log           : Present Board Tracer Number : SP33513621 "
  ...
          15_2    2015-06-09T03:07:08+09:00       critical        ""Flash disk can be replaced now. Status              : WARNING - PEER FAILURE Manufacturer        : Sun Model Number        : Flash Accelerator F80 PCIe Card Size                : 186GB Serial Number       : 11000101767 Firmware            : UIO6 Slot Number         : PCI Slot: 5; FDOM: 3 Cell Disk           : FD_15_dbm1cel05 Grid Disk           : Not configured Flash Cache         : Present Flash Log           : Present Board Tracer Number : SP33513621 "
  
  ii. Additional commands can be used to check as well the flash disk status:
  
  # cellcli -e list celldisk attributes name,status where disktype=flashdisk
       FD_00_dbm1cel05        normal
  ...
           FD_12_dbm1cel05        proactive failure
           FD_13_dbm1cel05        proactive failure
           FD_14_dbm1cel05        proactive failure
           FD_15_dbm1cel05        proactive failure
  
  iii. Check that these flash disks that are marked for replacement (FD12-FD15 in this example) are NOT caching griddisks (the cachedby attribute):
  
  # cellcli -e list griddisk attributes name,cachedby
           DATA_DBM1_CD_00_dbm1cel05     "FD_08_dbm1cel05, FD_11_dbm1cel05, FD_09_dbm1cel05, FD_10_dbm1cel05"
           DATA_DBM1_CD_01_dbm1cel05     "FD_00_dbm1cel05, FD_01_dbm1cel05, FD_03_dbm1cel05, FD_02_dbm1cel05"
  ...
           DATA_DBM1_CD_11_dbm1cel05     "FD_00_dbm1cel05, FD_01_dbm1cel05, FD_03_dbm1cel05, FD_02_dbm1cel05"
  
  iv. The status of the physicaldisk and luns will also not be normal:
  
  # cellcli -e list physicaldisk where disktype=flashdisk
  
           FLASH_1_0        normal
  ...
           FLASH_5_0        not present
           FLASH_5_1        proactive failure       complete
           FLASH_5_2        proactive failure       complete
           FLASH_5_3        proactive failure       complete
  
  # cellcli -e list lun where disktype=flashdisk
           1_0     1_0     normal
  ...
           5_0     5_0     failed
           5_1     5_1     warning - peer failure
           5_2     5_2     warning - peer failure
           5_3     4_3     warning - peer failure
  
  v. Validate on ASM that no rebalance operation is running. Otherwise, it has to wait for completion before replacing the Flash Card.
  
  IMPORTANT - In WriteBack cache mode, the Flash card can be replaced ONLY after the alert with message "Flash disk can be replaced now" has been generated and there is not rebalance running on ASM.
6. Run cellcli command to Inactivate all grid disks on the cell that needs to be powered down for maintenance. (this could take up to 10 minutes or longer)
  
  # cellcli
  ...sample ...
  CellCLI> ALTER GRIDDISK ALL INACTIVE
  GridDisk DATA_CD_00_dbm1cel01 successfully altered
  GridDisk DATA_CD_01_dbm1cel01 successfully altered
  ...
  GridDisk RECO_CD_02_dbm1cel01 successfully altered
  ...repeated for all griddisks...
7. Execute the command below and the output should show asmmodestatus='UNUSED' or 'OFFLINE' and asmdeactivationoutcome=Yes for all griddisks once the disks are offline and inactive in ASM.
  
  CellCLI> list griddisk attributes name,status,asmmodestatus,asmdeactivationoutcome
  DATA_CD_00_dbm1cel01 inactive OFFLINE Yes
  DATA_CD_01_dbm1cel01 inactive OFFLINE Yes
  ...
  RECO_CD_02_dbm1cel01 inactive OFFLINE Yes
  ...repeated for all griddisks...
8. Once all disks are offline and inactive, the customer may shutdown the Cell using the following command:
  
  # shutdown -hP now
  
  When powering off Exadata Storage Servers, all storage services are automatically stopped.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Confirm the PCIe slot needing replacement based on the output provided ("name" or "slotNumber" value) in the alert.

The Exadata Storage Server based on Sun Server X4-2L has six PCIe slots. They are numbered 1 through 6 with 1 nearest the Power Supplies, and 6 nearest the outside wall of the chassis (the onboard ports/connectors are located between slots 3 and 4). Slot locations for Flash F80 PCIe cards in Exadata Storage Servers are PCIe Slot 1, 2, 4 and 5.

The Sun Flash Accelerator F80 PCIe card does not have any field-serviceable parts, the FRU is the entire card unlike the previous Sun Flash Accelerator F20/F20M2 PCIe cards that previous Exadata versions used.

Slide out the server for maintenance. Do not remove any cables prior to sliding the server forward, or the loose cable ends will jam in the cable management arms.
Remove the AC power cords prior to removing the server’s top cover.
Remove the server’s top cover.
Rotate the PCIe card locking mechanism latch out to disengage the PCIe card for the slot that has failed.
Remove the PCIe card and place it on an anti-static mat.
Install the replacement PCIe card into the empty slot.
Rotate the PCIe card locking mechanism latch to secure the PCIe card into the slot.
Re-install the server’s top cover and reattach the AC power cords. ILOM will take up to 2 minutes to boot.
Slide the server back into the rack.
After ILOM has booted, power on the server by pressing the power button, and then connect to the server’s console.

To connect to the console through ILOM:

From the ILOM Web browser (preferred):
Press the "Launch” button next to "Remote Console" in the Actions tab on the initial Summary screen. This can also be launched by selecting the "Remote Control -> Redirection" selection on the left side menu.
From the ILOM CLI:

-> start /SP/console

OBTAIN CUSTOMER ACCEPTANCE
- WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

It is expected that the engineer stay on-site until the customer has given the approval to depart. The following commands are provided as guidance in case the customer needs assistance checking the status of the system following replacement. If the customer or the FSE requires more assistance following the physical replacement of the device, EEST/TSC should be contacted.

After replacing the Flash F80 PCIe card and updating its firmware if necessary, the Exadata Storage Server should boot up automatically. Once the Exadata Storage Server comes back online the cell services will start up automatically, however you will need to reactivate the griddisks as follows:

Check the status of the physicaldisk and luns of the replaced flash drive are now normal:

# cellcli -e list physicaldisk where disktype=flashdisk
FLASH_1_0 11000... normal
...
FLASH_5_0 11000... normal
FLASH_5_1 11000... normal
FLASH_5_2 11000... normal
FLASH_5_3 11000... normal

# cellcli -e list lun where disktype=flashdisk
1_0 1_0 normal
...
5_0 5_0 normal
5_1 5_1 normal
5_2 5_2 normal
5_3 5_3 normal

If the flash disks were offlined due to 'poor performance' and are still showing as 'poor performance' after replacement, then the luns may need to be deleted and recreated before they can be used again. Refer to Note 1306635.1 for specific instructions before continuing.
Verify the flashcache and flashlog are no longer showing degraded disks, and show status 'normal':

# cellcli -e list flashlog detail
name: dbm1cel01_FLASHLOG
...
degradedCelldisks:
effectiveSize: 512M
...
status: normal

# cellcli -e list flashcache detail
name: dbm1cel01_FLASHCACHE
...
degradedCelldisks:
...
size: 2.908935546875T
status: normal
Activate the griddisks:

# cellcli
…
CellCLI> alter griddisk all active
GridDisk DATA_CD_00_dbm1cel01 successfully altered
GridDisk DATA_CD_01_dbm1cel01 successfully altered
...
GridDisk RECO_CD_02_dbm1cel01 successfully altered
...etc...
Verify all disks show 'active':

CellCLI> list griddisk
DATA_CD_00_dbm1cel01        active
DATA_CD_01_dbm1cel01        active
...
RECO_CD_02_dbm1cel01        active
...etc...
Verify all grid disks have been successfully put online using the following command. Wait until 'asmmodestatus' is in status 'ONLINE' for all grid disks. The following is an example of the output early in the activation process.

CellCLI> list griddisk attributes name,status,asmmodestatus,asmdeactivationoutcome
DATA_CD_00_dbm1cel01 active ONLINE Yes
DATA_CD_01_dbm1cel01 active ONLINE Yes
...
RECO_CD_00_dbm1cel01 active SYNCING Yes
RECO_CD_01_dbm1cel01 active ONLINE Yes
...etc...

Notice in the above example that 'RECO_CD_00_dbm1cel01' is still in the 'SYNCING' process. Oracle ASM synchronization is only complete when ALL grid disks show ‘asmmodestatus=ONLINE’. This process can take some time depending on how busy the machine is, and has been while this individual server was down for repair. (Note: This operation uses Fast Mirror Resync operation - which does not trigger an ASM rebalance. The Resync operation restores only the extents that would have been written while the disk was offline.)
If WriteBack flashcache mode is enabled, then validate the cell has resumed caching to the flashdisks.
The following will show whether the mode is WriteBack or WriteThrough:

CellCLI> list cell attributes name,flashcachemode

In WriteBack mode, the following should show each griddisk being cached by all flash disks:

CellCLI> list griddisk attributes name,status,cachedby

A way to validate that data is dirty (i.e. only in flash disks and not yet on harddisks) run the following:

CellCLI> list metriccurrent fc_by_used,fc_by_dirty

On a busy system, metric "fc_by_dirty" should start increasing in value.

PARTS NOTE:

Refer to the Exadata Database Machine Owner's Guide Appendix C for part information.

Oracle Exadata X4-2 - Full Components List (https://mosemp.us.oracle.com/handbook_internal/Systems/Exadata_X4_2/components.html#SCSI)

Oracle Exadata X4-2 Storage Cell (X4-2L) - Full Components List (https://mosemp.us.oracle.com/handbook_internal/Systems/Exadata_X4_2_Storagecell/components.html)
Oracle Exadata Storage Expansion X4-2 - Full Components List (https://mosemp.us.oracle.com/handbook_internal/Systems/Exadata_X4_2/components.html)

REFERENCE INFORMATION:

Exadata Database Machine Documentation:

Exadata Database Machine Owner's Guide is available on the Storage Server OS image in /opt/oracle/cell/doc/welcome.html
http://amomv0115.us.oracle.com/archive/cd_ns/E13877_01/welcome.html
Sun Server X4-2L Server Product Library Documentation (includes Sun Server X4-2L Service Manual) http://docs.oracle.com/cd/E36974_01/index.html

How to Replace a Sun Server X4-2L PCIe Card (Doc ID 1592588.1 )
Exadata Storage Server gives a warning of unsupported firmware during boot if mixed Flash F80 firmware is installed (Doc ID 1529290.1)

Internal Only References:
- INTERNAL Exadata Database Machine Hardware Troubleshooting (Doc ID 1360360.1)

References

<NOTE:1188080.1> - Steps to shut down or reboot an Exadata storage cell without affecting ASM
<NOTE:1545103.1> - Replacing FlashCards or FDOM's when Griddisks are created on FlashDisk's
<NOTE:1306635.1> - Flash Disks may report 'Not Present' or 'Poor Performance' after FDOM/Flash Disk Replacement

Attachments

This solution has no attachment