Sun Network QDR InfiniBand Gateway Switch
 
Oracle Exalogic Elastic Cloud Software
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: SaND-CAP VCAP
 

In this Document

Goal

Solution

Applies to:

Goal

Enter the goal of the document. What does the customer want to accomplish?

How To Back Up and Restore Switch Settings for Sun Network QDR InfiniBand Gateway Switch.

Solution

DISPATCH INSTRUCTIONS
WHAT SKILLS DOES THE ENGINEER NEED:
Familiarity with the InfiniBand switch.

TIME ESTIMATE: 60 minutes

TASK COMPLEXITY: 0

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:
PROBLEM OVERVIEW
Collect backup configuration of the switch for later restoration of the configuration when required.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?

The InfiniBand switch will need to be up, running and reachable across the network. For restoring the configuration to the switch, the firmware should be at the same level as when the backup configuration was collected.

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE:

Backing up, restoreing, and verifing the replacement switch is the customers responcibility.

IMPORTANT:
If Enterprise Manager Ops Center (EMOC) was involved in the configuration of an Exalogic system components (nodes, IB switches, etc.) please consult with EEST Support for additional considerations to restore the configurations related to the new IB GW switch.

Backing Up and Restoring the InfiniBand Gateway Switch Settings.

The procedure for backing up and restoring switch settings can be done either through the switch Integrated Lights Out Manager (ILOM) or through a manual process.

Backing Up Settings on the InfiniBand Gateway Switch using ILOM:

The following procedure describes how to back up a switch using ILOM. The backup only needs to be done once after the switch has been initially configured with the correct settings.

1. Navigate to the switch ILOM URL in a browser. For example: http://dbm002-i1.us.example.com.
2. Log in as the ilom-admin user. The default password is ilom-admin.
3. Select the Maintenance tab.
4. Select the Backup/Restore tab.
5. Select the Backup operation and the Browser method.
6. Type in a passphrase. This is used to encrypt sensitive information, such as user passwords, in the backup.
7. Click Run, and save the resulting XML file in a secure location.
8. Login to the switch via a terminal shell as the root user. The default password is welcome1 or changeme.
9. Save the output from the 'version' command.
10. Save the output from 'ps -ef | grep open' command to determine whether OpenSM subnet manager daemon is already running on this switch. It may be intentionally disabled and not be running if this is a multi-rack of 4 or more full-racks, or if its connected to an Exadata.
11. Manually capture the following files not currently handled by the backup:

• /etc/opensm/opensm.conf
• /conf/bx.conf
• /conf/bxm.conf
• /conf/partitions.current
• /conf/smnodes

Restoring Settings to the InfiniBand Gateway Switch using ILOM:

The following procedure describes how to restore the settings on a switch using ILOM.

1. Run the version command, and ensure that the switch is at the right firmware level. If not, then upgrade the switch to the correct firmware level.
2. Navigate to the switch ILOM URL in a browser. For example: http://dbm002-i1.us.example.com.
3. Log in as the ilom-admin user. The default password is ilom-admin.
4. Select the Maintenance tab.
5. Select the Backup/Restore tab.
6. Select the Restore operation and the Browser method.
7. Click Browse, and select the XML file that contains the switch configuration backup.
8. Type in the passphrase that was used during the backup.
9. Click Run to restore the configuration.
10. Do the following steps to copy the file not captured by ILOM backup:

• Log in to the Sun Datacenter InfiniBand Switch 36 switch as the root user.
• Use the scp command to copy the opensm.conf file to the /etc/opensm/opensm.conf location on the switch. The opensm.conf file was captured during the previous backup operation above.
• Restore the /conf/bx.conf
• Restore the /conf/bxm.conf
• If the switch is runnning as a Subnetmanager:

Ensure the smnodes is created as follows:

sp->smnodes add {ip-address ipaddress ip-address ip-address}

Verify all switches configured as Subnet Managers smnodes is correct:

sp->smnodes list

• If OpenSM was previously running prior to restore, then restart OpenSM from the switch CLI using the following commands

sp-> disablesm

sp-> enablesm

• • If partitions were previously configured prior to restore, then resync the partitions by logging into the currently running Subnetmanager Master and doint the following:

Determine the current SM Master:

sp-> getmaster

Login to the Subnet Manager Master and sync the partition information:

sp-> smpartition start

sp-> smpartition commit

sp-> smpartition list active

 

On the newly resored switch:

sp-> smpartition list active

Ensure the partition information is syncronized by verifiing the version_number is the same and all the partition information matches.

 

Backing Up Settings on the InfiniBand Gateway Switch using CLI:

Logged into the switch as ilom-admin in another window:
-> cd /SP/config
/SP/config

-> set passphrase=welcome1
Set 'passphrase' to 'welcome1'

->  set /SP/config dump_uri=scp://root:welcome1@192.168.1.254/backup/cli_backup
Dump successful.
->

NOTE: 192.168.1.254 is a node configured with sshd that the backup will be scp's to.

Restoring Settings on the InfiniBand Gateway Switch using CLI:

-> cd /SP/config
/SP/config

-> set passphrase=welcome1
Set 'passphrase' to 'welcome1'

Back on the system I am logged in as ilom-admin:
-> set /SP/config load_uri=scp://root:welcome1@192.168.1.254/backup/cli_backup
Load successful.
->

NOTE: 192.168.1.254 is a node configured with sshd that the backup will be scp's from.

OBTAIN CUSTOMER ACCEPTANCE
- WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

In situations where the switch is replaced and we have restored the switch configuration, there is a need to verify the configuration and operation of the gateway switch.

1. Log in to the switch as the root user. The default password is welcome1 or changeme.
2. Verify switch configuration (hostname, ntp, dns, IP/Netmask information, gateway, ..etc.).
3. Verify switch operation by running the following commands:

• ibstat
• ibswitches
• ibhosts
• sminfo
• ibdiagnet -ls 10 -lw 4x
• ibdiagnet -r

REFERENCE INFORMATION:

Exalogic Machine Owner's Guide: https://docs.oracle.com/cd/E18476_01/index.htm
Sun Network QDR InfiniBand Gateway Switch Administration Guide - https://docs.oracle.com/cd/E19671-01/821-1187-11/821-1187-11.pdf
Sun Network QDR InfiniBand Gateway Switch Command Reference - https://docs.oracle.com/cd/E19671-01/821-1189-11/821-1189-11.pdf
Sun Network QDR InfiniBand Gateway Switch Service Manual - https://docs.oracle.com/cd/E19671-01/821-1188-11/821-1188-11.pdf
Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for the Sun Network QDR InfiniBand Gateway Switch - https://docs.oracle.com/cd/E19671-01/821-1545-11/821-1545-11.pdf

This solution has no attachment