ODA: SSH Error While Attempting To Use oakcli: "Error in authorized_keys" file has more than one line OR "failed to set up ssh for usr" grid or oracle or root

Asset ID:	1-72-2041882.1
Update Date:	2017-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2041882.1 : ODA: SSH Error While Attempting To Use oakcli: "Error in authorized_keys" file has more than one line OR "failed to set up ssh for usr" grid or oracle or root

Applies to:

Oracle Database Appliance X3-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X4-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.
SSH
oakcli
patching
authorized_keys

Symptoms

Problems with SSH while issuing an ODA OAKCLI command including:

Applying a core key that was generated and saved on node0

[root@oda1]# oakcli apply core_config_key /u01/.../Configkey.txt
INFO: Both nodes get rebooted automatically after applying the license
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed the reboot

Please enter the root password:

.....ERROR: Error in authorized_keys, file has more than one line on node 192.168.16.24 (see above text) at /opt/oracle/oak/lib/oaklib/apply line 156
While attempting to set up SSH user equivalence using the ./GridInst command for step 2 [root] or step 12 for users grid and oracle

INFO : Logging all actions in /opt/oracle/oak/onecmd/tmp/STEP-12-20150728155934.log and traces in /opt/oracle/oak/onecmd/tmp/STEP-12-20150728155934.trc
INFO : Loading configuration file /opt/oracle/oak/onecmd/onecommand.params...
INFO : Creating nodelist files...
INFO : 2015-07-28 15:59:35
INFO : Step 12 SetupSSHusers
INFO : Setting up ssh for users...
INFO : setting up ssh for user grid...
ERROR : failed to set up ssh for usr grid <<<<<<
INFO : Time in SetupSSHusers is 20 seconds.

Other possible scenarios where SSH problems can occur include:

Patching or upgrading the ODA
Including:
oakcli patch ...
oakcli upgrade ...
oakcli update ...
Creating or deleting a database or database home using oakcli
oakcli create database ...
oakcli create dbhome ...
oakcli delete dbhome ...
oakcli delete database ...

Note: There are other sources for SSH connection problems on the ODA

Document : 1568647.1 - How to reconfig ssh user equivalence Setup Post-Installation on ODA
Document : 1555600.1 - ODA GI Update Failed ERROR: Failed when creating tmp/cleaning ssh directory on Nod
Document : 1923446.1 - ODA: ERROR : Failed To Set Up Ssh For User Oracle
Document : 2003316.1 - ODA: SSH Errors During oakcli Patching, Upgrades, Deployment, Core Key Application or Creating Oracle Databases

Changes

Almost always after the successful deployment of the ODA

Cause

Using the ssh -v flag enabled debugging the connection problem in more detail
The debugging confirmed that there was a 3rd party product installed which we will call ACMEabc for this example
The example 3rd party application "ACMEabc" in this case alters the SSH configuration which is preventing the normal SSH communications between the ODA nodes leading to the oakcli command failing.

[root@oda1 ~]# ssh -v oc-oda1-nd2
OpenSSH_6.6, OpenSSL 0.9.8zc-fips 15 Oct 2014
debug1: Reading configuration data /root/.ssh/config <<
debug1: Reading configuration data /etc/ACMEabc/ssh/ssh_config <<< "ACMEabc" is a dummy name used for this example

The identification was in large part using SSH -V to debug the SSH connection.This revealed a non-Default path during the SSH connection.
In this case this also pointed to a 3rd party product:
The ODA does not install CENTRIFY by default: A user has added this product to tighten security for SSH

The ACTUAL 3rd Party PRODUCT INTRODUCED was CENTRIFY but it does not have to be this single product

root@oda1 ~]# ssh -v oda1
OpenSSH_6.6, OpenSSL 0.9.8zc-fips 15 Oct 2014
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 49: Applying options for *
...
...

"...Centrify Server Suite is an enterprise-class solution that secures Cloudera Enterprise Data Hub leveraging an organization’s existing Active Directory infrastructure to deliver access control, privilege management and user-level auditing..."

Working User

- In this example both root and oracle connections worked as expected even though centrify was in place

[root@oda1 ~]# ssh -v oda1
OpenSSH_6.6, OpenSSL 0.9.8zc-fips 15 Oct 2014
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 49: Applying options for *
debug1: Connecting to oda1 [192.168.123.33] port 22.
debug1: Connection established.                   << The connection here was redirected by the 3rd party Application but permanently_set_uid allows the ODA connection to go through
debug1: permanently_set_uid: 0/0                   <<<<<< Here is the difference we can detect using the SSH -v ( debug ) flag for the working and non-working user
debug1: identity file /root/.ssh/id_rsa type -1

...
...

MORE from strace
==================

[pid 79947] execve("/usr/share/centrifydc/bin/ssh", ["ssh", "oda1"], [/* 19 vars */]) = 0
[pid 79947] brk(0) = 0xe83000

Problem User

- GRID was the problem user for this case: However, this could be any combination of users including oracle, root or grid and is not a problem restricted to the GRID user.

GRID
====

[grid@oda1 ~]$ ssh -v oda1
OpenSSH_6.6, OpenSSL 0.9.8zc-fips 15 Oct 2014
debug1: Reading configuration data /home/grid/.ssh/config
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 49: Applying options for *
debug1: Connecting to oda1[192.168.123.45] port 22.
debug1: Connection established. << No permanently_set_uid: 0/0 referenced for the GRID user which is having the SSH problem
debug1: identity file /home/grid/.ssh/id_rsa type -1
...

MORE strace -f
=============
...
[pid 63047] execve("/usr/share/centrifydc/bin/ssh", ["ssh", "oda1"], [/* 20 vars */]) = 0
[pid 63047] brk(0) = 0x234e000
...

Solution

Remove or disable the 3rd party SSH application permanently or selectively for the duration of failing oakcli operations.

To confirm the problem compare the ssh -v outputs with working and non-working users and if needed compare with working vs. non-working ODA configurations

NOTE: the DEFAULT installation of the ODA would never see this specific problem.
The problem was introduced by installing a 3rd product intended to tighten security impacting key, specific ODA default users.

References

<NOTE:1555600.1> - ODA GI Update Failed ERROR: Failed when creating tmp/cleaning ssh directory on Nod
<NOTE:1923446.1> - ODA: ERROR : Failed To Set Up Ssh For User Oracle
<NOTE:2003316.1> - ODA: SSH Errors During oakcli Patching, Upgrades, Deployment, or Creating Oracle Databases
<BUG:20979272> - ODA INFRA UPDATE FAILS W/ERROR IN AUTHORIZED_KEYS WITH CENTRIFY AUTH INSTALLED
<NOTE:1638577.1> - ODA Troubleshooting ODA Core Key Generation and Application Problems
<BUG:16964721> - INFRA UPGRADE FROM 2.2 TO 2.6 FAILED DUE TO TMP/CLEANING SSH DIRECTORY ON NODE
<NOTE:2035218.1> - ODA ODAVP "oakcli apply core_config_key" Fails with "ERROR: Invalid Serial number, cannot set the core count"
<NOTE:1568647.1> - How to reconfig ssh user equivalence Setup Post-Installation on ODA

Attachments

This solution has no attachment