Oracle SuperCluster T5-8 Full Rack
 
Oracle Server X6-2L
 
Exadata X5-2 Eighth Rack
 
Exadata X5-2 Quarter Rack
 
Exadata X5-2 Hardware
 
Oracle SuperCluster M7 Hardware
 
Exadata X5-2 Full Rack
 
Exadata SL6 Hardware
 
Zero Data Loss Recovery Appliance X6 Hardware
 
Exadata X6-2 Hardware
 
Exadata X6-8 Hardware
 
Oracle SuperCluster T5-8 Half Rack
 
Exadata X4-8 Hardware
 
Exadata X5-2 Half Rack
 
Zero Data Loss Recovery Appliance X5 Hardware
 
Oracle Server X5-2L
 
Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 
Videos>Video_OVH
 
Microlearning>Video>ML-VID-VCAP
 

Applies to:

Goal

How to Replace an Oracle Server X5-2L, X6-2L or Exadata X5-2EF, X6-2EF CELL 12-Slot NVMe Disk Backplane.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:
Oracle Server X5-2L, X6-2L or Exadata X5-2EF, X6-2EF CELL Training

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 1

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Oracle Server X5-2L, X6-2L or Exadata X5-2EF, X6-2EF CELL 12-Slot NVMe Disk Backplane needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS.  Then power off the server and remove the AC power cords from the system.

A data backup is not a prerequisite but is a wise precaution.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

X5-2L Remove the Storage Drive Backplane for Twelve NVMe (Extreme Flash) Drive Systems:
https://docs.oracle.com/cd/E41033_01/html/E55031/goinp.html#scrolltoc

X6-2L Remove the Storage Drive Backplane for Twelve NVMe (Extreme Flash) Drive Systems:
http://docs.oracle.com/cd/E62172_01/html/E63689/goinp.html#scrolltoc

1. Log into the ILOM check the fruid container values and sync them if needed.

1. To avoid mismatched fruid values causing a failure after a disk backplane replacement the fruid data should be confirmed to have matching data in at least the Backup1 (MB) and Backup2 (PS0) containers so that the disk backplane will have it's container updated automatically after replacement. Go into restricted mode and use the showpsnc command to check this.  
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) x5-2l]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup1
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup2 (7)        | Backup2 (8)
   ------------------+-------------------+-------------------
   PPN                 33154888+1+1        33154888+1+1
   PSN                 1449NM7002          1449NM7002
   Product Name        ORACLE SERVER X5-2L ORACLE SERVER X5-2L
   [(restricted_shell) x5-2l]# exit
    
2. The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.  
   -> reset /SP
   Are you sure you want to reset /SP (y/n)? y
   Performing reset on /SP
    
3. After an ILOM reset if the Backup1 and Backup2 containers match then proceed with the following steps to replace the disk backplane. If these two containers do not match then DO NOT proceed with the replacement yet.
4. If the containers do not match you will need to use the copypsnc command from service or escalation mode to copy the data from the good container so that the Backup1 and Backup2 containers match (Primary is the DBP and we are about to replace this so it is not as important at this step). If you are unfamiliar with this process and require assistance please reference the steps for using copypsnc to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and contact the TSC if needed. How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1)
5. After the fruid data in the Backup1 and Backup2 containers have been confirmed to match proceed with the following steps.

2. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server to the maintenance position in the rack.
3. Attach an anti-static wrist strap.

3. Remove the Fan Assembly Door and all of the Fan Modules.

1. Using a No. 2 Phillips screwdriver, remove the two screws on each side of the chassis that secure the fan assembly door.
2. Slide the fan assembly door toward the rear of the server, then lift and remove the door from the chassis.
3. Using your thumb and forefinger, loosen the captive screw that secures the fan module in the chassis (use a No. 2 Phillips screwdriver if it is too tight to loosen with your fingers).
4. Grasp both the captive screw and the opposite end of the module and lift the fan module straight up and out of the chassis (do not rock it back and forth), and set it aside on an antistatic mat.

4. Remove all of the storage drives from the storage drive cage.

1. On each drive, push the latch release button to open the latch.
2. Grasp the latch and pull the drives far enough out so that they disengage from the backplane.
3. The drives do not need to be fully removed from the disk enclosure, but if they are then place the drives on an antistatic mat making sure to note the slot locations that the drives were removed from, so that they can be re-installed in their proper slots.

5. Disconnect the cables and remove the storage drive backplane.

1. Disconnect the twelve NVMe cables from the storage drive backplane.
2. Disconnect the two power cables and the auxiliary signal cable from the storage drive backplane.
3. Using a No. 2 Phillips screwdriver, loosen the two spring-mounted screws that secure the storage drive backplane to the chassis.
   Note -  Due to its location, the left spring-mounted screw might be difficult to remove with a standard No. 2 Phillips screwdriver. If you experience difficulty removing the screw, use a long No. 2 Phillips screwdriver or a torque driver with a long extension.
4. Lift the storage drive backplane up to release it from the standoff hooks.
5. Pull the storage drive backplane away from the standoff hooks and out of the chassis.
6. Place the storage drive backplane on an antistatic mat.

6. Install the new storage drive backplane and connect the cables.

1. Lower the storage drive backplane into the server, and position it to engage the standoff hooks.
2. Using a No. 2 Phillips screwdriver, install and tighten the two spring-mounted screws to secure the storage drive backplane to the chassis.
   Note -  Due to its location, the left spring-mounted screw might be difficult to install with a standard No. 2 Phillips screwdriver. If you experience difficulty installing the screw, use a long No. 2 Phillips screwdriver or a torque driver with a long extension.
3. Reconnect the two power cables and the auxiliary signal cable to the storage drive backplane.
4. Reconnect the twelve NVMe cables to the storage drive backplane.  Each NVMe cable contains a matching label that corresponds to the port on the Oracle PCIe NVMe switch card and the storage drive backplane connector.
1. PCIe slot 1 to right side disk backplane, PCIe NVMe Switch Card Ports to Storage Drive Backplane Connectors (port 0 - port D, port 1 - port E, port 2 - port F).
2. PCIe slot 2 to right side disk backplane, PCIe NVMe Switch Card Ports to Storage Drive Backplane Connectors (port 0 - port A, port 1 - port B, port 2 - port C).
3. PCIe slot 5 to left side disk backplane, PCIe NVMe Switch Card Ports to Storage Drive Backplane Connectors (port 0 - port D, port 1 - port E, port 2 - port F).
4. PCIe slot 6 to left side disk backplane, PCIe NVMe Switch Card Ports to Storage Drive Backplane Connectors (port 0 - port A, port 1 - port B, port 2 - port C).
5. Verify NVMe cable connections once power is restored.
   
   Note -  After replacing the storage drive backplane, the NVMe cable connection test procedure must be performed before booting the storage server operating system. This procedure ensures that NVMe PCIe cabling is correct prior to booting the operating system.

7. Re-install the storage drives into the storage drive cage.

1. Making sure to install the drives back into the same slots from which they were removed, align the drive to the drive slot.
2. Slide the drive into the bay until the drive is fully seated.
3. Close the drive latch to lock the drive in place.

8. Re-install the Fans.

1. Install the fan modules into the server.
2. For each module press down on the fan module and apply firm pressure to fully seat the fan module.
3. Using your thumb and forefinger, tighten the captive screw to secure each of the fan modules to the chassis. Then use a No. 2 Phillips screwdriver to tighten the screw an additional 1/4 turn to secure the fan module to the chassis.

9. Re-install the AC power cords and verify the NVMe cable connections.

1. Reconnect the AC power cords, power on the server, and enter the BIOS Setup Utility.  To enter the BIOS Setup Utility, press the F2 key (Ctrl+E from a serial connection) when prompted and while the BIOS is running the power-on self-tests (POST).
   Important - When NVMe cables are removed or replaced between the storage drive backplane and NVMe switch cards, you must perform the procedure in this section to confirm that all NVMe cable connections are correct. If NVMe cable connections are not correct, the storage server operating system should not be allowed to boot, as it could cause a problem with disk drive mapping.
2. Log into the ILOM CLI and enter restricted mode and run the NVMe cable connection test.
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) X5-2EF:~]# hwdiag io nvme_test
   HWdiag (Restricted Mode) - Build Number 94599 (Nov 17 2014, 10:59:38)
            Current Date/Time: Mar 23 2015, 18:27:15
       Checking NVME drive fru contents...
           checking fru on drive NVMe 0              OK
           checking fru on drive NVMe 1              OK
           checking fru on drive NVMe 2              OK
           checking fru on drive NVMe 3              OK
           checking fru on drive NVMe 4              OK
           checking fru on drive NVMe 5              OK
           checking fru on drive NVMe 6              OK
           checking fru on drive NVMe 7              OK
           checking fru on drive NVMe 8              OK
           checking fru on drive NVMe 9              OK
           checking fru on drive NVMe 10             OK
           checking fru on drive NVMe 11              OK
       NVME drives fru check:                        PASSED
   
       Checking NVME drive pcie links...
           checking pcie link on drive NVMe 0        OK
           checking pcie link on drive NVMe 1        OK
           checking pcie link on drive NVMe 2        OK
           checking pcie link on drive NVMe 3        OK
           checking pcie link on drive NVMe 4        OK
           checking pcie link on drive NVMe 5        OK
           checking pcie link on drive NVMe 6        OK
           checking pcie link on drive NVMe 7        OK
           checking pcie link on drive NVMe 8        OK
           checking pcie link on drive NVMe 9        OK
           checking pcie link on drive NVMe 10       OK
           checking pcie link on drive NVMe 11        OK
       NVME drives pcie link check:                  PASSED
   
       Checking NVME drive DSN...
           checking DSN on drive NVMe 0              OK
           checking DSN on drive NVMe 1              OK
           checking DSN on drive NVMe 2              OK
           checking DSN on drive NVMe 3              OK
           checking DSN on drive NVMe 4              OK
           checking DSN on drive NVMe 5              OK
           checking DSN on drive NVMe 6              OK
           checking DSN on drive NVMe 7              OK
           checking DSN on drive NVMe 8              OK
           checking DSN on drive NVMe 9              OK
           checking DSN on drive NVMe 10             OK
           checking DSN on drive NVMe 11              OK
       NVME drives DSN check:                        PASSED
   
       Checking NVME cabling...
           Cables associated with Switch Card 3 in PCIe Slot 6 verified
           Cables associated with Switch Card 2 in PCIe Slot 5 verified
           Cables associated with Switch Card 1 in PCIe Slot 2 verified
           Cables associated with Switch Card 0 in PCIe Slot 1 verified
       NVME cable check:                             PASSED
   
   NVME test PASSED
   [(restricted_shell) X5-2EF:~]#
3. If everything PASSED as shown above them continue to step 10.  If there are any fail statuses then the cable issue must be resolved before going past this step.  Remove the AC power cords again, and correct the cable issue.  After each cable change the above step 9 should be repeated until everything passes.

10. Re-install the fan assembly door.

1. Place the fan assembly door on the chassis and slightly over the fan assembly.
2. Slide the fan assembly door forward and under the lip of the forward top cover until it latches into place.
3. Using a No. 2 Phillips screwdriver, install and tighten the two screws on each side of the chassis.
4. Close the top cover fan assembly door.

11. Return the Server to operation.

1. Remove any anti-static measures that were used.
2. Return the server to it's normal operating position within the rack.
3. Re-install any data cables that were removed.
4. Power on server if not already powered on. Verify that the Power/OK indicator led lights steady on.

12. Set the system serial number/fruid data if needed.

1. The disk backplane is the primary fruid container in this server so when it is replaced you will normally need to fix the serial number information.
2. login to the ILOM as root and then enter the restricted shell to check the fruid values. Follow the example below to enter restricted shell and use the showpsnc command   
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) x5-2l:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 0000000000
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup1
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup2 (7)        | Backup2 (8)
   ------------------+-------------------+-------------------
   PPN                 33154888+1+1        33154888+1+1
   PSN                 1449NM7002          1449NM7002
   Product Name        ORACLE SERVER X5-2L ORACLE SERVER X5-2L
   [(restricted_shell) x5-2l:~]#
3. When the disk backplane is replaced the Primary fruid container will likely not match the Backup entries. If it does not you must enter escalation or service mode to fix it (if all three entries match this step is done).
4. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
5. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
6. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.
   -> cd /SP/users
   /SP/users
   -> create escuser
   Creating user...
   Enter new password: ********
   Enter new password again: ********
   Created /SP/users/escuser
   -> set escuser role=aucros
   Set 'role' to 'aucros'
   -> show escuser
   /SP/users/escuser
   Targets:
   ssh
   Properties:
   role = aucros
   password = *****
7. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.
   -> set /SP check_physical_presence=false
   Set 'check_physical_presence' to 'false'
   -> show /SP check_physical_presence
   /SP
   Properties:
   check_physical_presence = false
   
   -> exit
8. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.
   X5-2l login: escuser
   Password:
   
   Oracle(R) Integrated Lights Out Manager
   
   Version 3.2.4.36 r95733
   
   Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
   
   Warning: The system appears to be in manufacturing test mode.
   Contact Service immediately.
   
   Hostname: x5-2l
   
   -> cd /SP/users/ecsuser/escalation
   -> set SESSION mode=escalation                            
   Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
   Short form password is:  NOSE HAAG MED
   
   [(escalation_mode) X5-2l:~]#
9. Use the showpsnc command to confirm the current container values. Confirm that one of the backup containers has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid backup use the copypsnc command to write the good data from the backup to the primary container on the DBP. The following example shows copying from backup1 to the primary but you could also copy from backup2 if needed.
   
   [(escalation_mode) X5-2l:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 0000000000
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup1
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup2 (7)        | Backup2 (8)
   ------------------+-------------------+-------------------
   PPN                 33154888+1+1        33154888+1+1
   PSN                 1449NM7002          1449NM7002
   Product Name        ORACLE SERVER X5-2L ORACLE SERVER X5-2L
   [(escalation_mode) X5-2l:~]# copypsnc Backup1 Primary
   [(escalation_mode) X5-2l:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup1
   ------------------+-------------------
   PPN                 33154888+1+1
   PSN                 1449NM7002
   Product Name        ORACLE SERVER X5-2L
   
   Element           | Backup2 (7)        | Backup2 (8)
   ------------------+-------------------+-------------------
   PPN                 33154888+1+1        33154888+1+1
   PSN                 1449NM7002          1449NM7002
   Product Name        ORACLE SERVER X5-2L ORACLE SERVER X5-2L
   [(escalation_mode) X5-2l:~]# exit
   
   
10. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Backup1 Backup2" to copy backup1 to backup2) After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.
    
    -> reset /SP
    Are you sure you want to reset /SP (y/n)? y
    Performing reset on /SP
    ..........
    
    ***login as the root user again and check the fruid data***
    
    -> set SESSION mode=restricted
    
    WARNING: The "Restricted Shell" account is provided solely
    to allow Services to perform diagnostic tasks.
    
    [(restricted_shell) x5-2l]# showpsnc
    Primary: fruid:///SYS/DBP
    Backup 1: fruid:///SYS/MB
    Backup 2: fruid:///SYS/PS0
    
    Element           | Primary
    ------------------+-------------------
    PPN                 33154888+1+1
    PSN                 1449NM7002
    Product Name        ORACLE SERVER X5-2L
    
    Element           | Backup1
    ------------------+-------------------
    PPN                 33154888+1+1
    PSN                 1449NM7002
    Product Name        ORACLE SERVER X5-2L
    
    Element           | Backup2 (7)        | Backup2 (8)
    ------------------+-------------------+-------------------
    PPN                 33154888+1+1        33154888+1+1
    PSN                 1449NM7002          1449NM7002
    Product Name        ORACLE SERVER X5-2L ORACLE SERVER X5-2L
    [(restricted_shell) x5-2l]# exit
    
    -> cd /SP/users
    /SP/users
    -> delete escuser
    Are you sure you want to delete /SP/users/escuser (y/n)? y
    Deleted /SP/users/escuser
11. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

How to verify the Disk Backplane is working properly.

     1.  Log into ILOM to confirm if disk backplane status is working properly.

Sample

    2.  Check ILOM event log to see if any error related backplane.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality

REFERENCE INFORMATION:

Oracle Exadata Storage Server X5-2 Extreme Flash Documentation:
https://docs.oracle.com/cd/E41033_01/html/E55031/index.html

Oracle Exadata Storage Server X6-2 Extreme Flash Documentation:
http://docs.oracle.com/cd/E62172_01/html/E63689/index.html

Oracle Integrated Lights Out Manager (ILOM) 3.2 Documentation:
http://docs.oracle.com/cd/E37444_01/index.html

This solution has no attachment