"mammoth-reconfig add asr" Fails with "Error 11807" on BDA V4.1

Asset ID:	1-72-1982023.1
Update Date:	2015-12-18
Keywords:

Solution Type Problem Resolution Sure

Solution 1982023.1 : "mammoth-reconfig add asr" Fails with "Error 11807" on BDA V4.1

Applies to:

Big Data Appliance X4-2 Full Rack - Version All Versions and later
Big Data Appliance X5-2 In-Rack Expansion - Version All Versions and later
Big Data Appliance X4-2 In-Rack Expansion - Version All Versions and later
Big Data Appliance X4-2 Starter Rack - Version All Versions and later
Big Data Appliance X4-2 Hardware - Version All Versions and later
Linux x86-64

Symptoms

Adding ASR through "./mammoth-reconfig add asr" fails on BDA V4.1 with the following error:

# ./mammoth-reconfig add asr
INFO: Logging all actions in /opt/oracle/BDAMammoth/bdaconfig/tmp/bdanode01-<timestamp>.log and traces in /opt/oracle/BDAMammoth/bdaconfig/tmp/bdanode01-<timestamp>.trc
INFO: This is the install of the primary rack
INFO: Checking if password-less ssh is set up
INFO: Executing checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
SUCCESS: Executed checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
INFO: Executing checkSSHAllNodes.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
INFO: Error code 1 when executing checkSSHAllNodes.sh on host bdanode02
INFO: Error is : Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
INFO: Error code 1 when executing checkSSHAllNodes.sh on host bdanode03
INFO: Error is : Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
INFO: Error code 1 when executing checkSSHAllNodes.sh on host bdanode04
INFO: Error is : Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
...
INFO: Error code 1 when executing checkSSHAllNodes.sh on host bdanode0n
INFO: Error is : Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
INFO: Password-less SSH not set up, skipping reboot check
INFO: Reading component versions from /opt/oracle/BDAMammoth/bdaconfig/COMPONENTS
INFO: Creating nodelist files...
Enter the value for ASR_HOST [Default: xxx]:
Enter the value for ASR_PORT [Default: 162]:
Enter the value for ASR_SERVER_USER [Default: root]:

Please Check the values you entered for the ASR parameters

ASR_HOST = xxx
ASR_PORT = 162
ASR_SERVER_USER = root

Are these values correct (y/n): y
Enter password for user root on machine exadatabck
Enter password:
Enter password again:
INFO: Creating environment.pp file ...
INFO: Cloudera Manager password is not set in config
INFO: Using values in saved params file
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents
INFO: Setting up ASR on all nodes. This will take some time ...
INFO: Puppet script successfully sent to node bdanode01
INFO: Puppet script successfully sent to node bdanode02
INFO: Puppet script successfully sent to node bdanode03
INFO: Puppet script successfully sent to node bdanode04
...
INFO: Puppet script successfully sent to node bdanode0n
INFO: Pinging puppet agents ..
ERROR: Puppet agent run on node bdanode01 had errors. List of errors follows

************************************
Error [11807]: (//bdanode01.domain.xx/Puppet) Could not retrieve catalog from remote server: Error 400 on SERVER: 'undef' from right operand of 'in' expression is not of a supported type (string, array or hash) at /opt/oracle/BDAMammoth/puppet/manifests/site.pp:8 on node bdanode01.domain.xx
************************************

INFO: Also check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode01-<timestamp>.log

Note: This applies to new BDA V4.2 installs as well.

Cause

The error is due to the following two causes:

1. /opt/oracle/BDAMammoth/puppet/manifests/site_setup_asr.pp missing the line ""import "targetnodes""

2. One of the "mammoth-reconfig add asr" scripts, /opt/oracle/BDAMammoth/bdaconfig/MammothSteps/SetupASR.pm, searches for ASR binaries in path /opt/SUNWswasr/bin/asr on the ASR manager server which is no longer valid for the latest ASR version. The correct path is now /opt/asrmanager/bin/asr.

The following bug has been filed for this issue: Bug 20569739 - MAMMOTH-RECONFIG ADD ASR IS FAILING WITH ERROR [11807].

The ASR 5.3 Quick Install Guide, has details on the ASR Manager path. According to the new July 2015 ASR Deployment Guide, the ASR Manager path has been changed to: /opt/asrmanager/bin from: /opt/SUNWswasr/bin/asr.

Solution

1. Edit the file /opt/oracle/BDAMammoth/puppet/manifests/site_setup_asr.pp. At the top of the file, you will see the lines:

import "environment"
import "settings"

Add the following line immediately below the above two lines.

import "targetnodes"

2. Edit the file named /opt/oracle/BDAMammoth/bdaconfig/MammothSteps/SetupASR.pm, and replace the string /opt/SUNWswasr/bin/asr with /opt/asrmanager/bin/asr. In other words in /opt/oracle/BDAMammoth/bdaconfig/MammothSteps/SetupASR.pm:

Change:

/opt/SUNWswasr/bin/asr

To:

/opt/asrmanager/bin/asr

Another workaround until the fix is available is to create a symbolic link on the ASR Manager between the old depreciated directory and the new one:

# ln –s /opt/asrmanager/bin /opt/SUNWswasr/bin

This workaround should work for cases where a user agrees to create the symbolic link on the ASR Manager. It will allow the Mammoth configuration of ASR to complete. Then repeat the failing mammoth command.

References

<BUG:20569739> - MAMMOTH-RECONFIG ADD ASR IS FAILING WITH ERROR [11807]

Attachments

This solution has no attachment