How to generate a DCLI Group file for, and collect diagnostics on, Exadata or SuperCluster IB Switches.

Asset ID:	1-71-1608918.1
Update Date:	2014-01-15
Keywords:

Solution Type Technical Instruction Sure

Solution 1608918.1 : How to generate a DCLI Group file for, and collect diagnostics on, Exadata or SuperCluster IB Switches.

Applies to:

Exadata Database Machine V2 - Version All Versions to All Versions [Release All Releases]
Linux x86-64
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Exadata & SuperCluster Engineered Systems

Goal

To create & test a dcli group file for IB switches, then use same to establish SSH authority & gather common IB diagnostics.

Oracle Exadata & SuperCluster Engineered Systems contain multiple InfiniBand (IB)
Switches, but do not (by default) have a suitable dcli group file listing same.

Solution

The following could (in principle) be run from any node containing 'dcli', and as any user,
but for our purposes should be run as 'root' from the 1st DB node, where the rest of the
dcli *_group files already exist, in the /root directory:

case $(uname -p) in sparc ) ibswitches | cut -d\" -f2 | \
cut -d. -f1 | egrep -iv 'adm|ilom|localhost' | awk '{ print $(NF - 1) }' | \
sort -u | tee ibsw_group ;; * ) ibswitches | cut -d\" -f2 | \
cut -d. -f1 | egrep -iv 'adm|ilom|localhost' | awk '{ print $NF }' | \
sort -u | tee ibsw_group ;; esac

# Double-check that the above output shows the expected switch
# node names (and edit to correct if not), then test further with:

for IB in `cat ibsw_group` ; do case $(uname) in Linux )
ping -c 1 $IB | cut -d: -f1 -s ;; SunOS ) ping $IB ;; esac ; done

# Above should show 1 line for each switch of the form (on Linux):
# <nn> bytes from <FQDN> (<IP>)

# Linux example:

64 bytes from dmnnnx8sw-ib1.us.oracle.com (10.141.xxx.nnn)

# or (on Solaris):
# <name> is alive

# Solaris example:

orlscsw-ib1 is alive

# If not, then double-check the switch nodenames in the group file, and the name/IP address resolution for same.
# Otherwise, proceed with:

dcli -g ibsw_group -l root -k -t

# above will setup the SSH equivalence for the root user between the DB Node and the switches,
# prompting for passwords interactively; supply same, then retest with:

dcli -g ibsw_group -l root uname

# Above should respond with Linux for each switch, w/no password prompts.
# Note: IB switches run Linux, 32-bit CentOS 5.2 (2.6.27.13-nm2 i686).
# Note also that SSH _should_ be setup already on Exadata, and trouble-shooting
# SSH is beyond the scope of this Note, but see other Notes in References below.
# If above works, then so should this:

dcli -g ibsw_group -l root -t '{ DL=$(printf \
"%60s" | tr " " "=") ; echo $DL ; version | grep DCS ; echo $DL ; \
ibstatus | grep [es]: ; echo $DL ; setsmpriority list | grep smp ; \
echo $DL ; env_test | tail -1 ; echo $DL ; showunhealthy ; echo $DL ; \
listlinkup ; echo $DL ; }' | tee /tmp/IB_Switch_Info.log

# Attach the file IB_Switch_Info.log to the SR.
# Optionally run (if requested):

dcli -c $(tail -1 ibsw_group) -l root '{ DL=$(printf \
"%60s" | tr " " "=") ; echo $DL ; sminfo ; echo $DL ; getmaster -l ; echo \
$DL ; ibcheckerrors ; echo $DL ; ibnetdiscover ; echo $DL ; ibdiagnet -skip \
dup_guids -pm ; echo $DL ; }' | tee /tmp/IB_Switch_CheckErr.log

# Then also attach the file IB_Switch_CheckErr.log to the SR.

References

<NOTE:1053498.1> - Network Diagnostics information for Oracle Database Machine Environments
<NOTE:372795.1> - How to Configure SSH for User Equivalence to install the Grid Control Agent 10.1 to 10.2.0.4
<NOTE:1437596.1> - SSHD Connection Errors After Upgrading To Exadata Version 11.2.3.1.0
<NOTE:1458480.1> - SSH long wait between retries, or user account locked out.
<NOTE:787205.1> - Setting up DCLI Utility on Oracle DB host

Attachments

This solution has no attachment