Node in Oracle Big Data Appliance with LDAP Authentication Setup takes a very Long Time to Boot/Startup

Asset ID:	1-72-1666562.1
Update Date:	2016-10-11
Keywords:

Solution Type Problem Resolution Sure

Solution 1666562.1 : Node in Oracle Big Data Appliance with LDAP Authentication Setup takes a very Long Time to Boot/Startup

Applies to:

Big Data Appliance X3-2 Hardware - Version All Versions and later
Linux x86-64

Symptoms

Rebooting a Oracle Big Data Appliance (BDA) node with LDAP Authentication setup takes about 20+ hours to complete.

Boot up seems to take a long pause at three separate places:

Starting mdmonitor
...
Starting system message bus (It hangs the longest after this one)
...
After Starting yum-updatesd

Below errors are noticed in /var/log/messages file:

nss_ldap: failed to bind to LDAP server ldaps://<servername>/: Can't contact LDAP server

<node name> kernel: RDS: rds_bind() could not find a transport, load rds_tcp or rds_rdma?

Cause

During boot vnic_wait service failed and thus virtual network interface card (VNIC's) didn't get started. Network will not come up properly if VNIC's are not started thus leading to failure or LDAP authentication calls.

Solution

Increase sleep time in /etc/init.d/vnic_wait script thus allowing VNIC's to startup correctly. On Oracle Big Data Appliance, V2.5/ higher for 2.* releases and V3.1/higher for 3.* releases have sleep time set to 3 in /etc/init.d/vnic_wait script by default.

Editing /etc/init.d/vnic_wait script will effect all run levels, as all runlevel files are symlinks to /etc/init.d/vnic_wait.

Slow reboot issue may occur on all nodes on the cluster so use dcli to increase sleep time on all nodes in the BDA Cluster

1) Log as root into node01 of the primary rack.

2) Save a copy of /etc/init.d/vnic_wait script

3) Copy /etc/init.d/vnic_wait to /tmp

# cp /etc/init.d/vnic_wait /tmp

4) Edit /tmp/vnic_wait script to increase sleep from 1 to 3 (if needed to a higher value ) in start () routine.

For example after change

fi
        sleep 3
    done
    failure $"only $sys_count/$dev_count devices found"

5) Copy the edited /tmp/vnic_wait file to all nodes in the cluster into /etc/init.d directory.

dcli -C ls -l /etc/init.d/vnic_wait
dcli -C -f /tmp/vnic_wait -d /etc/init.d/vnic_wait
dcli -C ls -l /etc/init.d/vnic_wait

References

<NOTE:1476008.1> - Using DCLI on Oracle Big Data Appliance

Attachments

This solution has no attachment