Zero Data Loss Recovery Appliance X6 Hardware
 
Exadata X5-2 Eighth Rack
 
Exadata X6-2 Hardware
 
Exadata X5-2 Hardware
 
Exadata X5-2 Full Rack
 
Exadata X5-2 Quarter Rack
 
Zero Data Loss Recovery Appliance X5 Hardware
 
Exadata X5-2 Half Rack
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Exadata FRU only CAP

Applies to:

Goal

 Canned Action Plan for replacing a Motherboard in compute node in an Exadata Database Machine [X5-2/X6-2]

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?: Exadata Trained

TIME ESTIMATE: 120 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW:A server in an Exadata Database Machine requires the motherboard to be replaced. This procedure is specific to
Exadata X5-2/X6-2 systems based on Oracle Server X5-2/X6-2.

Connectivity to the rack will depend on the customer's access requirements. The following procedure partially requires serial connection
and network access to ILOM which assumes using a laptop attached to the Cisco management switch. If no port is available in a full
rack, then temporarily disconnect a port used for another host's ILOM (e.g. port 2). If the customer does not allow login access to the
host ILOM, then they will need to run the commands given below.

When connecting to ILOM via serial cable remember that the baud rate is 9600 for replacement boards. This will get changed during
the post-install procedure to the Exadata default which is 115200 for installed boards.

To make room for needed text MB = Motherboard

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

Pre-Install Steps:

1. Backup ILOM Settings.
      Assuming the ILOM is not the reason for the replacement of the system MB, then take a current backup of the ILOM SP
      configuration using a browser under “ILOM Administration → Configuration Management” tab on the left menu list.

This can also be done from the ILOM CLI as follows:-

2. Obtain the correct Serial Numbers required.

     (a) Make a note of the System Serial Number from the front label of the server.
     (b) Make a note of the Rack Master Serial Number from the front label of the rack (left-side vertical wall, half way up the rack).

3. If the system is not down already due to whatever problem is causing the MB to be replaced, then have the customer DBA shut the
node down.

(a) For Extended information on this section, check MOS Note 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS
Services and Cell/Compute Nodes On An Exadata Configuration. (https://support.oracle.com/epmos/faces/ui/km
/SearchDocDisplay.jspx?id=1093890.1&type=DOCUMENT )

For a documentation reference, in the Exadata Owner’s Guide (E13874), use the section of chapter 7 titled “Non-Emergency Power
Procedures” section “Powering Off Oracle Exadata Rack” sub-section “Powering off Database Servers” available on the customer's
cell server image in the /opt/oracle/cell/doc.

http://amomv0115.us.oracle.com/archive/cd_ns/E13877_01/doc/doc.112/e13874/maintenance.htm#autoId18

(b) The Customer should shutdown CRS services prior to powering down the DB node,

if running OVM then go to section "For Compute Node running OVM - for non-OVM proceed as follows:

Shutdown crs

i. As root user do the following to stop crs and disable autostart of crs on reboot: 

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.

In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

ii. Validate CRS is down cleanly. There should be no processes running.

(c)Check to see if CoD (Capacity on Demand) is configured.

Issue the following command and make a note of the total number of active physical cores.If the count is 36 then CoD is not in use.

If the compute node which requires the motherboard replacement is down then run this command on another compute node in the same cluster.

Record the result.

Check to see if IaaS is configured. If ON this will have to enabled after rebooting.If a blank line is retruned then IaaS is not configured.

(d) The customer or the field engineer can now shutdown the server operating system:

Linux:

(d) The field engineer can now slide out the server for maintenance. Remember to disconnect the power cords before opening the top of the server .

 For Compute Node running OVM proceed as follows:

If there are any concerns engage EEST engineer.

The customer should perform the following:

(a) See what user domains are running (record result )

Connect to the management domain (domain zero, or dom0).

  This is an example with just two domains and the management domain Domain-0

connect to each domain  using the command

where domainname would be dm01db01vm01 or dm01db01vm02 if using the above examples.

Shut down any instances of crs ,refer to the example above in previous section "shutdown crs" in all user domains

Press CTRL+] to disconnect from the console.

(b)Shutdown all user domains from dom0

(c) See what user domains are running (should be only Domain-0)

(d) Disable user domains from auto starting during dom0 boot after motherboard has been replaced. 

 (e) Check to see if CoD (Capacity on Demand) is configured.

Issue the following command and make a note of the total number of active physical cores.If the count is 36 then CoD is not in use.

If the compute node which requires the motherboard replacement is down then run this command on another compute node in the same cluster.

Record the result.

Check to see if IaaS is configured.If ON this will have to be enabled after rebooting.If a blank line is returned then IaaS is not configured.

(f) The customer or the field engineer can now shutdown the server operating system:

 (g)The field engineer can now slide out the server for maintenance. Remember to disconnect the power cords before opening the top of the server .

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

Note: The removal/insertion CPU tool is new for the Ivy Bridge M3 product lines. If you have not used this new tool before please make yourself familiar before
attempting to use on-site. The tool is not intuitive so reference the service manual before attempting this service action.

Reference links for Service Manual:
X5-2 DB’s: ( http://docs.oracle.com/cd/E41059_01/html/E48312/napsm.html#scrolltoc )

Physical Replacement Steps:

1. Replace the MB as per MOS Note "TO BE ENTERED" migrating existing CPUs, DIMMs, PCI Cards and risers.

2. Carefully follow the port numbers on the cables when re-attaching so they are not reversed. It is easiest to plug cables in while the server is in the fully extended
maintenance position.

3. Do not power up the system yet, just ILOM.

Post-Installation Steps:

1. Update the Serial Number on the new MB, to that of the server chassis. This is REQUIRED in order for ASR to continue to work on the unit, and is required
for all servers that are part of Exadata racks that may have a future Service Request, whether ASR is configured now or not.

These platforms use the Top Level Indicator (TLI) feature in ILOM to perform the MB serial number update automatically. In certain circumstances this may
not work correctly and will need manually corrected. For more information on TLI and restricted shell please refer to the following 2 MOS notes for these
systems:-

TLI MOS Note 1280913.1
Restricted Shell MOS Note 1302296.1

NOTE: The serial numbers of each server can be found at the front on the left hand side.

(a) Connect to ILOM via serial port, and login as “root” with default password “changeme”.
(b) Enter Restricted Mode.

(c) Review the current PSNC containers with “showpsnc” command:

If the replacement has the correct product serial number, in all 3 containers including “/SYS/MB” then skip to step 2 of the post-replacement procedures. If
the replacement does not have the product serial number populated correctly, then “exit” out of restricted shell mode and continue:

(d) Where there is at least one container which still contains valid TLI information (usually the primary disk backplane DBP0), a service mode command
copypsnc can be used to update the product serial number.

i. Login as root and create escalation mode user with service role:

ii. Gather “version”, “show /SYS” and “show /SP/clock” outputs needed for generating the service mode password:

iii. Generate a service mode password using “http://modepass.us.oracle.com/” Login is via Oracle Single-Sign-On. Example output of the tool is:

iv. Logout of root and log back in as 'sunny' user that you created, and enter Service mode:

The copypsnc command produces no output upon success.

vi. After running copypsnc, the service processor should be rebooted.
 

 
vii. Log in again as 'root' user with default password 'changeme' and verify the SN is now populated correctly using 'show /SYS' and 'showpsnc' as
shown above.

viii. Remove the 'sunny' user:

 
If there are any issues with programming the serial number with “copypsnc” then an escalation mode password and instructions will need to be provided by
the TSC x86 engineer assigned to the SR.

2. Re-flash the ILOM/BIOS to the correct levels required for Exadata if required.

Exadate X5-2 requires minimum image 12.1.2.1.0 this image automatically checks the version of ILOM/BIOS on the motherboard and will attempt to correct this if required .

Before power up it is necessary to configure the ILOM network settings , obtain the ILOM network values from the customer or the previously saved SP.config file . Login to the ILOM then "-> cd /SP/network" and apply the settings.If this step is not performed the ILOM will fail to re-flash.

i. Power up system using front button or from ILOM "-> start /SYS"
ii. During the boot validation phase, CheckHWnFWProfile will run and determine the ILOM version, and give a WARNING if it is not correct ,it will then attempt to reflash ,which will result in a reboot.

 Login as root to the node after it is booted, and run the following:

3. Restore the backed up SP configuration done during the pre-installation steps.

(a) Using a browser under Maintenance Tab or from ILOM cli:-

If SP backup was not possible check with customer for network information & use another ILOM within the rack for general settings. The primary specific
setup for Exadata are:

i. Serial Baud rate is 115200 for external and host
-> show -l all /SP/serial
speed = 115200 for "external" and "host"
if this is not 115200 (ILOM default is 9600, Exadata default is 115200), then they should be set:

ii. /SP system_identifer is set to the appropriate rack type string and master Rack Serial Number. This is critical for ASR deployments. The Master Rack
Serial number can be obtained top left inside the cabinet or from show /SP on any other ILOM. The string should be of the following format:
X5-2 - “Exadata Database Machine X5-2 <Rack SN>”
For Example:

iii. /SP hostname is setup
iv. /SP/network settings
v. /SP/alertmgmt rules that may have been previously setup by ASR or cell configuration
vi. /SP/clock timezone, datetime, and /SP/clients/ntp NTP settings
vii. /SP/clients/dns Name service settings
viii. root account password. If the root password has not been provided you can have the customer do this, or do this manually:

(b) Reset the ILOM under the Maintenance Tab or from ILOM cli:

(c) Check you can login to all interfaces and ILOM can be accessed using a browser and ssh from another system on the customer's management network.

4. Power-on the host server, and go into BIOS setup and check BIOS settings against EIS checklist, in particular make sure USB is first in boot order if this is a
Storage Cell (the original USB stick should have been moved from the old board to the new board), and check date and time is correct. Use ILOM cli to set
the boot device to BIOS "-> set /HOST boot_device=bios" or press F2 (Ctrl-E) during BIOS at the right time to get into BIOS setup menu.

5. Power-on and boot the system, monitoring the graphics java console through ILOM (or local video if there is a crash cart available). As the system boots the

hardware/firmware profile will be checked, and either a green “Passed” will be displayed, or a red “Warning” that something with the hardware or firmware does
not match what is expected. If the check passes, then everything is correct and up, and the boot will continue up to the OS login prompt. If the check fails, then
the issue being flagged should be investigated and rectified before continuing.

6. Additional OS checks:-

(a) Verify the network interfaces have correctly picked up the new MAC addresses of the new system board:

Verify that the management network is working:

Verify that the ILOM management network is working:

Verify that all memory is present in Linux. Sun Server X5-2 DB nodes have 256GB.

- this may vary depending on BIOS version.

Verify the disks are visible and online:

Verify the hardware logical volume is correctly set up:

Verify HW Profile is operating correctly.

If there are any errors, they will need to be corrected.

(b) Verify the InfiniBand connections are up and actively seen in the fabric:

If possible to login to DB01, then check InfiniBand connections are ok by running the following from DB01:

(options to verify-topology may be required depending on configuration)
If not possible for security reasons, then on this local node, verify the IB connection status with:

(c) Verify server functionality :

(d) If dcli is setup for password-less SSH, then the SSH keys need to be updated for new mac address. The customer should be able to do this using their
root password.

6. [Eighth Rack Only] For DB Nodes in Eighth Rack configurations, the configuration requires limiting CPU cores for licensing as follows. Refer to MOS Note
1538561.1 for more details if required.

To confirm if this is an Eighth rack config,ask the customer to login to a storage cell which is part of the this cluster and issue the following command:

If a blank line is displayed then this is not an Eighth rack config.Above shows a Eighth rack has been detected

(a) Verify the current configuration with the following command:

For an eighth rack configuration, you should see 18 cores enabled. If that's what you see, then there are no configuration changes needed and the rest of this
procedure should not be used.

(b) If the output shows 36 cores enabled as shown above, we need to change the configuration with the following command:

Reboot the host:

(c) After the node reboots, verify the changes are now made:

If the compute node is configured for CoD restore the config if required.Check the reported value is the same as previously recorded.

 If the value is incorrect then adjust the core count

Verify the pending number of active physical cores using the following command: 

 Now reboot the compute node.

If IaaS is configured:

Verify that all physical cores are active using the following command:

Enable IaaS

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

This is for compute nodes NOT running OVM proceed as follows,if running OVM see later section "For compute node running OVM"

You can now hand the system back to the customer DBA to check all ASM or DB CRS services can be brought up and are online before obtaining sign-off. This step
may take more than 10 minutes to complete based on the current load on the database. See detailed information below. If the customer DBA requires assistance
beyond this, then you should direct them to callback the parent SR owner in EEST.

DB Node Startup Verification:

1. Startup CRS and re-enable autostart of crs. After the OS is up, the Customer DBA should validate that CRS is running. As root execute:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.
In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.
Example output when all is online is:

2. Validate that instances are running:

It should return a record for the ASM instance and a record for each database.

For Compute Node running OVM

If the customer requires assistance please ask them to contact EEST engineer or parent case owner.

Once the compute node has booted ,re-enable user domains to autostart during Domain-0 boot.

Startup all user domains that are marked for auto start

See what user domains are running (compare against result from previously collected data)

if any not auto-started then Startup a single user domain

 Check that crs has started in user domains ,refer to previous section "DB Node Startup Verification"

PARTS NOTE:

7098505 System Board Assembly for Exadata X5-2 DB Nodes (Sun Server X5-2)

REFERENCE INFORMATION:

Service Manual's:
X5-2 DB’s: ( http://docs.oracle.com/cd/E41059_01/html/E48312/napsm.html#scrolltoc )
X5-2 Motherboard Replacement Procedure: MOS Note TO BE INSERTED when available.
MOS Note 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.
MOS Note ID 1188080.1 Steps to shut down or reboot an Exadata storage cell without affecting ASM

MB Serial Number Reprogramming:
TLI MOS Note 1280913.1
Restricted Shell MOS Note 1302296.1
Exadata Database Machine Maint Guide: http://amomv0115.us.oracle.com/archive/cd_ns/E50790_01/doc/doc.121/e51951/toc.htm
EIS Checklist: http://eis.us.oracle.com/checklists/pdf/Exadata-X5-2-8B.pdf

This solution has no attachment