Engineered Systems - IB Switch Upgrade via patchmgr Fails With "[FAIL] Error(s) encountered during execution"

Asset ID:	1-72-2329763.1
Update Date:	2018-04-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2329763.1 : Engineered Systems - IB Switch Upgrade via patchmgr Fails With "[FAIL] Error(s) encountered during execution"

Applies to:

Exadata X6-2 Hardware - Version All Versions and later
Exadata X5-2 Hardware - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
Exadata X4-2 Hardware - Version All Versions and later
Exadata X7-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Upgrading an Infiniband switch via patchmgr in an Engineered System fails.

The patchmgr.log file shows an error stack similar to the following but does not indicate the cause of the failure.

..........<SNIP>..........

[1510857559][2017-11-16 10:40:05 -0800][FAIL][/EXAVMIMAGES/patching/26635229/Infrastructure/12.2.1.1.2/ExadataStorageServer_InfiniBandSwitch/26027495/patch_12.2.1.1.2.170714/upgradeIBSwitch.sh][checkNUpgradeIBSwitchSW][3694][DISPLAY] Overall status

[1510857559][2017-11-16 10:40:05 -0800][INFO][/EXAVMIMAGES/patching/26635229/Infrastructure/12.2.1.1.2/ExadataStorageServer_InfiniBandSwitch/26027495/patch_12.2.1.1.2.170714/upgradeIBSwitch.sh][upgradeIBSwitch_doIt][] [FAIL] Error(s) encountered during execution.

----- InfiniBand switch update process ended 2017-11-16 10:40:05 -0800 -----

The /var/log/installfw.log from the switch being patched shows the following error stack indicating a login problem during the copying of the firmware bundle to the switch.

[16 Nov 10:39:58][main:264]: ==============Started Installfw============
[16 Nov 10:39:58][getShelfManagerDetails:1380]: Executing command: version 2>>/tmp/installfw.stderr
[16 Nov 10:39:58][getShelfManagerDetails:1393]: SUN DCS 36p version: 2.2.5-3

..........<SNIP>.........
[16 Nov 10:39:58][checkIfPingable:372]: Executing ping -c 3 172.30.247.94 2>&1 >/tmp/pingoutput
[16 Nov 10:40:00][generic_run_command:99]: /usr/local/bin/xcp -r 100 -t 60 -T 3600 scp://root@172.30.247.94//EXAVMIMAGES/patching/26635229/Infrastructure/12.2.1.1.2/ExadataStorageServer_InfiniBandSwitch/26027495/patch_12.2.1.1.2.170714/sundcs_36p_repository_2.2.6_2.pkg - 2>/tmp/fetcherrors | package ls -u /tmp/magnumfw_repository/magnumfw_repository -l > /tmp/magnumfw_repository/magnumfw_repository/headers.DSC 2>/tmp/unpackerrors
[16 Nov 10:40:00][getPkgFile:703]: ERROR: - 1

[16 Nov 10:40:00][getPkgFile:712]: Error: Login denied

[16 Nov 10:40:00][getPkgFile:719]: ERROR: 3 - 7

[16 Nov 10:40:00][generic_run_command:99]: rm -rf /tmp/magnumfw_repository/magnumfw_repository 2>>/tmp/installfw.stderr
[16 Nov 10:40:00][fw_ilom_exit:204]: 3 - 7 - 7
[16 Nov 10:40:00][fw_ilom_exit:205]: ==============Completed Installfw============

Cause

This failure can be caused by either wrong/missing SSH keys or SSH not allowing root login.

Solution

Checking root SSH Login

1. On the node where the firmware bundle (patch) is located, check that the root account can login via ssh. This is controlled by an option in the file /etc/ssh/sshd.config. Ensure the line:

PermitRootLogin no

is either commented out or changed to

PermitRootLogin yes

If this option is set to no or is not commented out, then go to #2.

If this option is already commented out or set to yes, go to the Checking SSH Keys section below.

2. Enable root ssh login by commenting the line or changing the value to yes and saving the changes to the file.

3. Retry the upgrade of the switch.

If the upgrade succeeds, go to #4.

If the upgrade still fails, go to the Checking SSH Keys section below.

4. Patch the remaining switches.

5. If you made any changes to the /etc/ssh/sshd_config file in #2, revert the changes.

6. Once all switches are upgraded and any changes made to the sshd_config file have been reverted, stop here. You are finished.

Checking SSH Keys

1. As root, log in to the IB switch that failed to upgrade.

2. Attempt to copy a small file from the location of the firmware bundle to the switch using xcp.

For example, if the firmware bundle (patch) is on a node at IP address 123.45.67.89, this example copies the /etc/hosts file from it to the /tmp directory on the switch

/usr/local/bin/xcp -r 100 -t 60 -T 3600 -v scp://root@123.45.67.89//etc/hosts /tmp

The command should succeed with verbose output similar to the following:

     * About to connect() to 123.45.67.89 port 22 (#0)
     * Trying 123.45.67.89... * connected
     * Connected to 123.45.67.89 (123.45.67.89) port 22 (#0)
     * SSH authentication methods available: publickey,password
     * Using ssh public key file /root/.ssh/id_rsa.pub
     * Using ssh private key file /root/.ssh/id_rsa
     * Initialized SSH public key authentication
     * Authentication complete
     * Closing connection #0

If there are any login denied error messages, go to #3.

     * About to connect() to 123.45.67.89 port 22 (#0)
     * Trying 123.45.67.89... * connected
     * Connected to 123.45.67.89 (123.45.67.89) port 22 (#0)
     * SSH authentication methods available: publickey,password
     * Using ssh public key file /root/.ssh/id_dsa.pub
     * Using ssh private key file /root/.ssh/id_dsa
     * Authentication failure
     * Closing connection #0
     * Login denied
     Error: Login denied

If there are any other errors except login denied errors, to to #12.

3. Check that the key file listed in the output from #2 actually exists. Using our example from #2 above:

ls -l /root/.ssh/id_dsa.pub

If the key exists, go to #4.

If the key does not exist, go to #5.

4. Make a back up of the key file:

mv /root/.ssh/id_dsa.pub /root/.ssh/id_dsa.pub.bak

5. Create the key.

For RSA keys - ssh-keygen -t rsa

For DSA keys - ssh-keygen -t dsa

6. On the node where the firmware bundle is stored, make a backup copy of the authorized keys file:

cp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.orig

7. Add the newly created key to the authorized keys file on the node where the firmware bundle is stored:

cat /root/.ssh/id_dsa.pub | ssh root@123.45.67.89 "cat >> /root/.ssh/authorized_keys"

8. Repeat #2.

If the test copy succeeds, go to #9.

If the test copy fails, go to #10.

9. Retry the upgrade of the switch.

If the switch upgrade succeeds, stop here. You are finished.

If the switch upgrade still fails, go to #10.

10. If a backup of the key file on the switch was made in #4, revert the backup.

mv /root/.ssh/id_dsa.pub.bak /root/.ssh/id_dsa.pub

11. On the node where the firmware bundle is stored, restore the back up copy of the authorized keys file:

cp /root/.ssh/authorized_keys.orig /root/.ssh/authorized_keys

12. Collect the following data:

The output of the failed copy test in #8.
The /var/log/installfw.log file from the switch.
The upgradeIBSwitch.log from the node where patchmgr is run.

13. Open a Service Request (SR) with Oracle Support.

14. Attach the data collected in #12 to the SR.

References

<BUG:26974460> - SAAS DB LCLDX0037: IB SWITCH UPGRADE FAILED WITH LOGIN DENIED

Attachments

This solution has no attachment