Exadata X7-2 Hardware
 
Zero Data Loss Recovery Appliance X7 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

Applies to:

Goal

How to Replace an Exadata X7-2 Compute Node Server Disk Backplane.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED:
Exadata X7-2 Training

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS

PROBLEM OVERVIEW: An Exadata X7-2 Compute Node Server Disk Backplane needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

IMPORTANT NOTE TO TSC ENGINEER: CUT & PASTE the “CUSTOMER ACTIVITY” sections of the Pre-Replacement and Post-Replacement steps into a SR Note and ensure the customer is aware to do these steps prior to the scheduled field engineer activity, and during and after the replacement activity.

CUSTOMER ACTIVITY:

Shutdown of the database node is required prior to the part replacement:

If running Linux or Solaris native - follow Steps 1 to 9 of MOS Note:
How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

If running OVM then follow MOS Note:
How to Shutdown and Startup Exadata database nodes running OVM (Doc ID 2367609.1)

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

Prepare the Server for Service

The customer should have already prepared the server and powered it off.  If not, provide them the instructions in the previous section.

1. Log into the ILOM check the fruid container values and sync them if needed. To avoid mismatched fruid values causing a failure after a disk backplane replacement the fruid data should be confirmed to have matching data. The Disk Backplane is the primary container so the Backup1 (MB) and Backup2 (PS0) must have valid values that are the same, in order for the replacement disk backplane to be updated to the correct values automatically.

Go into ILOM restricted mode and use the showpsnc command to check this:

The above example shows a system with all three containers properly in sync. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync. 

2. After an ILOM reset if the Backup1 and Backup2 containers match then proceed with the following steps to replace the disk backplane. If these two containers do not match then DO NOT proceed with the replacement yet.  Contact TSC for further assistance.

If the containers do not match you will need to use the "copypsnc" command from service or escalation mode to copy the data from the good container so that the Backup1 and Backup2 containers match (Primary is the disk backplane and we are about to replace this so it is not as important at this step). If you are unfamiliar with this process and require assistance please reference the steps for using "copypsnc" to fix the serial number detailed in the "How to update product serial number on systems which implement TLI functionality (Doc ID 1280913.1)" and "How to access service mode and escalation mode on ILOM 3.x and later platforms (Doc ID 1019946.1)". After the fruid data in the Backup1 and Backup2 containers have been confirmed to match proceed with the following steps.

3. Extend the server to the maintenance position

4. Disconnect the power cords from the power supplies.

5. Attach an anti-static wrist strap to your wrist and to a metal area on the chassis or the rack.

6. Remove the server top cover. Use a Torx T10 screwdriver to unlock the release button latch.

Removing the Faulted Disk Backplane

1. Lift up the super capacitor to release the capacitor from the reclosable fasteners that secure it to the chassis.
    Set the super capacitor aside (leaving the super capacitor cable attached) and out of the way of the disk backplane chassis space.
2. Pull all storage drives out far enough to disengage them from the disk backplane.

3. Disconnect the SAS cables from the disk backplane.
4. Disconnect the disk backplane data cable and the temperature sensor cable from the disk backplane.
5. Disconnect the disk backplane power cable from the disk backplane.
6. Using a Torx T10 screwdriver, loosen the three captive screws that secure the disk backplane to the chassis.
7. Lift the backplane up and tilt it toward the rear of the server to free it from the hooks and right-side metal bracket, and lift it out of the server.
8. Place the disk backplane on an antistatic mat.

Installing the Disk Backplane

1. Install the disk backplane into the server so that it engages the three metal hooks and also sits behind the metal tab located on the right-side chassis wall.
2. Using a Torx T10 screwdriver, tighten the three storage drive backplane captive screws clockwise until the disk backplane is secure.
3. Reconnect the disk backplane power cable to the disk backplane.
4. Reconnect the disk backplane data cable and temperature sensor cable to the disk backplane.
5. Reconnect the SAS cables to the disk backplane.
6. Re-secure the super capacitor by pressing down on the capacitor to secure it to the re-closable fastener in the chassis.
7. Re-install all of the storage drives into the server making sure that they are installed into their original locations, and fully seated and latched.

Return the Server to Operation

1. Install the server top cover. Use a Torx T10 screwdriver to lock the release button latch.
2. Reconnect the power cords to the server power supply and connect any other cables to their original locations.
3. Return the server to the normal rack position.
4. Once the power cords have been re-attached and the ILOM has booted you will see a slow blink on the green LED for the server. Power on the server by pressing the power button on the front of the unit.
5. Connect to the server console via the ILOM and monitor the boot.
      By default the ILOM serial console displays the primary console output.
      In the event of unexpected boot behavior, it is advisable to connect to both ILOM serial and ILOM graphics consoles at the same time and monitor.

6. Check and set the system serial number/fruid data:

a. Login to the ILOM as root and then enter the restricted shell to check the psnc values. Follow the example below to enter restricted shell and use the showpsnc command-

The above example shows a system with the Primary container not in sync after DBP replacement. If the output from the system does not show all of the containers with matching values then you should reset the SP and then re-check the values again. An ILOM reset will attempt to auto-populate the matching values if one container is out of sync.  

If after the ILOM reset all three entries match, this step is done. If they do not match, the containers will need manually programmed - contact the TSC for further assistance.

Manual serial number identity programming steps:

1. If the containers don't match you must enter escalation or service mode to fix it.
   
2. Contact the TSC to request an escalation password (service mode will work also if just the copypsnc command ends up needing to be used, if the setpsnc command is needed escalation mode is required. setpsnc is not covered in this procedure).
3. Provide your TSC contact the output from the following ILOM commands- "version", "show /SYS product_serial_number", and "show /SP/clock". If the product_serial_number information does not give good output then provide the showpsnc output that was seen in step b above as well.
4. The TSC will provide an escalation password that is made up of 32 short words. Follow the example below to create a new user with the 'Service' role assigned. The Service role is required to access service or escalation modes. In the following example we will create a user named 'escuser' with the service role.
   -> cd /SP/users
   /SP/users
   -> create escuser
   Creating user...
   Enter new password: ********
   Enter new password again: ********
   Created /SP/users/escuser
   -> set escuser role=aucros
   Set 'role' to 'aucros'
   -> show escuser
   /SP/users/escuser
   Targets:
   ssh
   Properties:
   role = aucros
   password = *****
5. Set the check_physical_presence to false and then exit from the ILOM so that you can login as the newly created user.
   -> set /SP check_physical_presence=false
   Set 'check_physical_presence' to 'false'
   -> show /SP check_physical_presence
   /SP
   Properties:
   check_physical_presence = false
   
   -> exit
6. Login using the escuser login and enter escalation mode using the password that was provided by the TSC.
   exa1dbadm01 login: escuser
   Password:
   
   Oracle(R) Integrated Lights Out Manager
   
   Version 4.0.0.22 r120818

   Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

   Warning: HTTPS certificate is set to factory default.

   Hostname: exa1dbadm01-ilom

   -> cd /SP/users/ecsuser/escalation
   -> set SESSION mode=escalation                            
   Password:**** **** **** **** **** *** *** **** **** **** **** **** **** **** **** **** *** *** **** *** **** **** **** *** **** **** *** **** *** *
   Short form password is:  NOSE HAAG MED
   
   [(escalation_mode) exa1dbadm01-ilom:~]#
7. Use the showpsnc command to confirm the current container values. Confirm that one of the backup containers has a serial number (the value on the PSN line) that matches the system serial number. The system serial number can be checked by comparing to the serial number RFID tag on the front left hand side of the server. After confirming that there is a valid fruid backup use the copypsnc command to write the good data from a backup to the primary container on the DBP. The following example shows copying from backup1 to the primary but you could also copy from backup2 if needed.
   
   [(escalation_mode) exa1dbadm01-ilom:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 7338405             7338405             7338405
   PSN                 0000000000          1735XC3004          1735XC3004
   Profile             0x00010000          0x00010000          0x00010000
   Product Name        ORACLE SERVER X7-2  ORACLE SERVER X7-2  ORACLE SERVER X7-2
   RFID SN             341A583DE5800000000232F8 341A583DE5800000000232F8 341A583DE5800000000232F8
   [(restricted_shell) exa1dbadm01-ilom:~]# copypsnc Backup1 Primary
   [(escalation_mode) exa1dbadm01-ilom:~]# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 7338405             7338405             7338405
   PSN                 1735XC3004          1735XC3004          1735XC3004
   Profile             0x00010000          0x00010000          0x00010000
   Product Name        ORACLE SERVER X7-2  ORACLE SERVER X7-2  ORACLE SERVER X7-2
   RFID SN             341A583DE5800000000232F8 341A583DE5800000000232F8 341A583DE5800000000232F8
   [(restricted_shell) exa1dbadm01-ilom:~]# exit
   
   
8. At this point if all of the fruid containers match and have the correct serial number data this step is done. If more than one of the fruid containers had non-valid entries then the copypsnc command should be used to copy over the valid data to the other container that is not valid. (ie. "copypsnc Backup1 Primary" to copy Backup1 to Primary) After confirming all fruid data is correct reset the ILOM to confirm that the fruid data persists through a reboot and remove the escalation user if needed.
   
   -> reset /SP                     
   Are you sure you want to reset /SP (y/n)? y
   Performing reset on /SP
   ..........
   
   ***login as the root user again and check the fruid data***
   
   -> set SESSION mode=restricted
   
   WARNING: The "Restricted Shell" account is provided solely
   to allow Services to perform diagnostic tasks.
   
   [(restricted_shell) exa1dbadm01-ilom:# showpsnc
   Primary: fruid:///SYS/DBP
   Backup 1: fruid:///SYS/MB
   Backup 2: fruid:///SYS/PS0
   
   Element           | Primary           | Backup1           | Backup2
   ------------------+-------------------+-------------------+-------------------
   PPN                 7338405             7338405             7338405
   PSN                 1735XC3004          1735XC3004          1735XC3004
   Profile             0x00010000          0x00010000          0x00010000
   Product Name        ORACLE SERVER X7-2  ORACLE SERVER X7-2  ORACLE SERVER X7-2
   RFID SN             341A583DE5800000000232F8 341A583DE5800000000232F8 341A583DE5800000000232F8
   [(restricted_shell) exa1dbadm01-ilom:~]# exit
   
   -> cd /SP/users
   /SP/users
   -> delete escuser
   Are you sure you want to delete /SP/users/escuser (y/n)? y
   Deleted /SP/users/escuser
9. If trouble is encountered during any of the steps of accessing escalation mode and fixing the fruid containers please contact the TSC for assistance.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

FIELD SERVICE ENGINEER and CUSTOMER ACTIVITY:

1. Verify all expected hardware is visible to the server and the fault is cleared. Assistance from the customer for server login access will be required.

2. Verify there are no outstanding faults in ILOM:

3. Verify all the expected disk devices are present. For Compute Nodes there should be a single volume:

4. Verify all expected logical drives are present and state 'Optl' (Optimal). For Compute Nodes there should be a single RAID5 volume:

5. Verify there are no outstanding alerts in the Database Node:

6. Re-enable and restart the Database services:

If running Linux or Solaris native - follow Steps 11 to 14 of MOS Note:
How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

If running OVM then follow MOS Note:
How to Shutdown and Startup Exadata compute nodes running OVM (Doc ID 2367609.1)

PARTS NOTE:

7341145 [F] 8-Slot Disk Backplane

REFERENCE INFORMATION:

Oracle Exadata Database Machine Maintenance Guide: https://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-database-servers.htm#DBMMN22020

Oracle Server X7-2 Documentation https://docs.oracle.com/cd/E72435_01/index.html

How to shutdown the Exadata database nodes and storage cells in a rolling fashion so certain hardware tasks can be performed. (Doc ID 1539451.1)

How to Shutdown and Startup Exadata compute nodes running OVM (Doc ID 2367609.1)

This solution has no attachment