Exadata X3-2 Eighth Rack
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

In this Document

Goal

Solution

Exadata Storage Cell Procedures - version 11.2.3.2.1 and lower

Exadata Storage Cell Procedures - version 11.2.3.3.0 and higher

Exadata Database Server Procedures

Troubleshooting

References

Applies to:

Goal

Following a restore or rescue of a database server or storage cell in an Exadata Database Machine eighth rack configuration, some steps are required to restore the proper configuration. This note provides the necessary steps to complete after a successful restore or rescue of a server in an eighth rack. 

Solution

 If a storage cell or database server in an eighth rack is reimaged, restored, or rescued, the steps here should be followed to ensure that the server remains configured properly. These steps should be followed after a complete restore or rescue of the node and after the node is added to the configuration and verified as fully functional again. Once those steps are complete, continue with the procedures here to ensure a proper eighth rack configuration. 

This procedure will also need to be employed if machine has a system board replacement.

Exadata Storage Cell Procedures - version 11.2.3.2.1 and lower

For Exadata Storage Cells running version 11.2.3.2.1 or lower, it is important to update or confirm that both CPU and storage updates are made. The storage cell may need to be rebooted more than once to complete the procedure. 

1. Copy the /opt/oracle.SupportTools/resourcecontrol utility from another database server or storage cell to the restored/recovered storage cell in /opt/oracle.SupportTools/resourcecontrol. All nodes should use the same version of this utility, so if it is updated on one node, all nodes should be updated.
2. Ensure proper permissions on the utility by running:
   
   (root)# chmod 740 /opt/oracle.SupportTools/resourcecontrol 
    
3. Verify the current configuration with the following command:
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -show
   [INFO] Validated hardware and OS. Proceed.
   [SHOW] Number of cores active per socket: All
   [SHOW] Number of harddisks active - 12.
   [SHOW] Number of flashdisks active - 16.
   (root)#
    
4. For an eighth rack configuration, you should see 6 cores enabled, 6 harddisks, and 8 flashdisks enabled. If that's what you see, then there are no configuration changes needed and the rest of this procedure should not be used.
5. If the output shows All cores, 12 harddisks, and 16 flashdisks enabled as shown above, we need to change the configuration with the following commands:
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -disk 6
   (root)# /opt/oracle.SupportTools/resourcecontrol -core 6
   (root)# reboot
    
6. After the node reboots, verify the changes are now made:
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -show
   [INFO] Validated hardware and OS. Proceed.
   [SHOW] Number of cores active per socket: 3
   [SHOW] Number of harddisks active - 6.
   [SHOW] Number of flashdisks active - 8.
   (root)#
    

Exadata Storage Cell Procedures - version 11.2.3.3.0 and higher

For Exadata Storage Cells running version 11.2.3.3.0 and higher, we must ensure that the storage cell is in the eighth rack configuration. 

1. Ensure your storage cell is configured as an eighth rack:
   
   (root)# cellcli -e list cell attributes eighthrack 
    
   If the output is either blank or false, then your cell is not configured as part of an eighth rack. If the output is "true," then you can skip this section. 
2. Modify the cell to configure it as an eighth rack cell:
   
   (root)# cellcli -e alter cell eighthRack=true
     

Exadata Database Server Procedures

For Exadata Database Servers, it is important to update or confirm that CPU updates are made. The database server may need to be rebooted more than once to complete the procedure.

1. Copy the /opt/oracle.SupportTools/resourcecontrol utility from another database server or storage cell to the restored/recovered database server in /opt/oracle.SupportTools/resourcecontrol. All nodes should use the same version of this utility, so if it is updated on one node, all nodes should be updated.
2. Ensure proper permissions on the utility by running:
   
   (root)# chmod 740 /opt/oracle.SupportTools/resourcecontrol
    
3. For Solaris database servers only, obtain the latest onecommand patch (found listed in <Note 888828.1> under the OneCommand heading). In a temporary staging directory on the Solaris database server, unzip the patch, then unpack the tar file in the patch. In the resulting directory, the file solaris-ubiosfiles.tar.gz will be found. That file needs to be unpacked on the node that was restored. See the example below (the p16663996_112321_Generic.zip file is the latest onecommand download at the time of writing):
   
   (root)# unzip -q p16663996_112321_Generic.zip -d /tmp/onecmdtmp 
   (root)# mkdir -p /tmp/onecmdtmp/stage
   (root)# tar xf /tmp/onecmdtmp/onecmd.tar -C /tmp/onecmdtmp/stage
   (root)# tar zxf /tmp/onecmdtmp/stage/solaris-ubiosfiles.tar.gz -C /
   (root)#
    
4. Verify the current configuration with the following command:
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -show
   [INFO] Validated hardware and OS. Proceed.
   [SHOW] Number of cores active per socket: All
   (root)#
    
5. For an eighth rack configuration, you should see 8 cores enabled. If that's what you see, then there are no configuration changes needed and the rest of this procedure should not be used.
6. If the output shows All cores enabled as shown above, we need to change the configuration with the following commands (on Solaris, use "reboot -p"):
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -core 8
   (root-Linux)# reboot
   (root-Solaris)# reboot -p      ### on Solaris, use reboot -p
7. After the node reboots, verify the changes are now made (this example is from a Solaris database server node - results are reported as 4 cores per socket, so 8 cores total):
   
   (root)# /opt/oracle.SupportTools/resourcecontrol -show
   [INFO] This is a Solaris DB node.
   [INFO] Validated hardware and OS. Proceed.
   [SHOW] Number of cores active per socket: 4
   (root)#

Troubleshooting

If during the course of the procedures there is an error like this one:

It is possible this is due to unpublished bug 16087908. The resolution is pending, but in the meantime, rebooting the node once or twice often clears this problem. Follow <Note 1638657.1> for help resolving this issue.

References

This solution has no attachment