Creating Second Database Using OEDA Fails with "OCMD-02624: Error While Executing Command {0}.java.lang.reflect.InvocationTargetException"

Asset ID:	1-72-2194997.1
Update Date:	2016-10-26
Keywords:

Solution Type Problem Resolution Sure

Solution 2194997.1 : Creating Second Database Using OEDA Fails with "OCMD-02624: Error While Executing Command {0}.java.lang.reflect.InvocationTargetException"

Applies to:

Exadata X6-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Exadata X6-2 env (can occur on any version of Exadata hardware)

Initial configuration (install/config of grid /database) has been implemented - CRS and database are up and running

Second database home has been installed using OEDA (Oracle Exadata Deployment Assistant)

Creating a second database in the new home using OEDA (install.sh) fails with OCMD-02624

/u01/mydb/linux-x64# ./install.sh -cf /u01/mydb/ExadataConfigurations_sun/sun-cluster.xml -s 15
Initializing
Executing Create Databases
Creating databases...
Execution Exception in future get
OCMD-02624: Error while executing command {0}.java.lang.reflect.InvocationTargetException
ERROR:
Errors occured.
Error: Errors occured...
Collecting diagnostics...
Errors occurred. Send /u01/mydb/linux-x64/WorkDir/Diag-161018_xxxxxx.zip to Oracle to receive assistance.

ERROR:
Errors occured...
Error: Errors occured...
Errors occured, exiting...

Step15_Create_Databases_161018_224839.out
++++++++++++++++++++++++++++++++++++++++++++++++

...

2016-10-18 22:48:55,792 [FINE][l-1-thread-1][ FileUtils:480] File /u01/mydb/linux-x64/WorkDir/oracle01.example.com_cluster_interconnect.dat exists, removing file /u01/mydb/linux-x64/WorkDir/oracle01.example.com_cluster_interconnect.dat
2016-10-18 22:48:55,793 [FINE][l-1-thread-1][ RemoteFileUtils:1623] Copying file /u01/mydb/linux-x64/WorkDir//u01/mydb/linux-x64/WorkDir/oracle01.example.com_OLTP_X2_2_112040.dbt to dest /u01/app/oracle/product/11.2.0.4/xksapp/assistants/dbca/templates//OLTP_X2_2_112040.dbt on nodes [[oracle01.example.com, oracle02.example.com]] as user oracle11g
2016-10-18 22:48:55,796 [FINE][ OCMDThread][ CommonUtils:362] Executing IPv4 ping on oracle02.example.com with command = /bin/ping -c1 oracle02.example.com on host: oracle01.example.com
2016-10-18 22:48:56,297 [FINE][l-1-thread-1][ RunCommand:750] Got kommand output 0 String is <Node oracle02.example.com is pingable> ret = <0>
2016-10-18 22:48:58,785 [FINE][ OCMDThread][ OcmdException:108] Stack trace...
2016-10-18 22:48:58,786 [FINE][ OCMDThread][ OcmdException:145] OcmdException from node oracle01.example.com return code = 2 output string: Auth fail Auth fail stack trace = java.lang.Throwable
at oracle.onecommand.escommon.common.OcmdException.ocmdException(OcmdException.java:105)
at oracle.onecommand.escommon.common.OcmdException.<init>(OcmdException.java:56)
at oracle.onecommand.commandexec.SesChannel.SessionPool.doSessionConnect(SessionPool.java:541)
at oracle.onecommand.commandexec.SesChannel.SessionPool.getJschSession(SessionPool.java:439)
at oracle.onecommand.commandexec.SesChannel.SessionPool.getJschSession(SessionPool.java:699)
at oracle.onecommand.commandexec.utils.SecureFtp.doSftp(SecureFtp.java:484)
at oracle.onecommand.commandexec.utils.RemoteFileUtils.sftpPutFile(RemoteFileUtils.java:1525)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.onecommand.commandexec.utils.CallableReflectionMethod.runMethodInParallel(CallableReflectionMethod.java:121)
at oracle.onecommand.commandexec.utils.CallableReflectionMethod.call(CallableReflectionMethod.java:70)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Info in /var/log/secure at the time of error
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Oct 18 22:48:40 sahisk01 sshd[357757]: Connection from 100.100.0.102 port 34425
Oct 18 22:48:40 sahisk01 sshd[357757]: Accepted password for root from 100.100.0.102 port 34425 ssh2
Oct 18 22:48:40 sahisk01 sshd[357757]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 18 22:48:43 sahisk01 sshd[357757]: subsystem request for sftp
Oct 18 22:48:44 sahisk01 sshd[357757]: subsystem request for sftp
Oct 18 22:48:56 sahisk01 sshd[361463]: Set /proc/self/oom_score_adj to 0
Oct 18 22:48:56 sahisk01 sshd[361463]: Connection from 100.100.0.102 port 34443
Oct 18 22:48:56 sahisk01 sshd[361463]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=oracle01.example.com user=oracle11g
Oct 18 22:48:58 sahisk01 sshd[361463]: Failed password for oracle11g from 100.100.0.102 port 34443 ssh2
Oct 18 22:48:58 sahisk01 sshd[361464]: Received disconnect from 100.100.0.102: 3: com.jcraft.jsch.JSchException: Auth fail

Testing ssh/scp to node itself and remote node work without error

Changes

Ct (re)set/change password for oracle user (oracle11g) after initial deployment of GRID/DATABASE through passwd command

# passwd oracle11g

Cause

The password stored in OEDA at initial deployment is different from one changed.

The install step tries to ssh to node itself as oracle(oracle11g) user from root, then fails as it uses password stored in OEDA configuration

Solution

If you want to set up a custom password (change password) for the oracle and grid users after initial deployment, then you need to run genPasswordHash.sh utility to update the OEDA configuration file with the desired password.
The same password will be used for both oracle and grid users.

To reset all non-root user password in OEDA configuration (to the one changed after initial configuration)

./genPasswordHash.sh -non-root

Document : 2163751.1 - OEDA Deployment - Generating new or non-default hash password for root and non-root usersDelete Reference

******INTERNAL *******

here are two use cases for using the genPasswordHash.sh
utility:

1. If your Exadata nodes are configured with a custom password for the root user, then you need to run this utility to update the OEDA configuration file with the custom root password.
This enables install.sh to log into the Exadata nodes using the correct password.

2. If you want to set up a custom password for the oracle and grid users, then you need to run this utility to update the OEDA configuration file with the desired password.
The same password will be used for both oracle and grid users.

Make sure you enter valid passwords when prompted. Passwords are encrypted in the OEDA configuration file.

Usage: ./genPasswordHash.sh [-root | -non-root | -all]

Options:
-root Update the OEDA configuration file with the root password.
The same password for root will be used on ALL nodes.
This is the default option.

-non-root Update the OEDA configuration file with the password set for the grid and oracle users.

The same password will be set for both users on all nodes.

-all This option is a combination of the -root and -non-root options.

References

<NOTE:2163751.1> - OEDA Deployment - Generating new or non-default hash password for root and non-root users

Attachments

This solution has no attachment