bdachecknet fails on "test admin network resolve and reverse resolve" in BDA V4.5 OL6 Mammoth Upgraded Cluster

Asset ID:	1-72-2209154.1
Update Date:	2017-03-02
Keywords:

Solution Type Problem Resolution Sure

Solution 2209154.1 : bdachecknet fails on "test admin network resolve and reverse resolve" in BDA V4.5 OL6 Mammoth Upgraded Cluster

Applies to:

Big Data Appliance X5-2 Starter Rack - Version All Versions and later
Big Data Appliance Integrated Software - Version 4.5.0 to 4.5.0 [Release 4.5]
Linux x86-64

Symptoms

On a BDA v4.5 ol6 Mammoth upgraded cluster bdachecknet may report incorrect name resolution for the admin network:

# bdachecknet

bdachecknet: do basic sanity checks on /opt/oracle/bda/network.json
bdachecknet: passed
bdachecknet: checking for networkexpansion.json
bdachecknet: ping test private infiniband ips (bondib0 40gbs)
bdachecknet: passed
bdachecknet: ping test admin ips (eth0 1gbs)
bdachecknet: passed
bdachecknet: test admin network resolve and reverse resolve
bdachecknet: ip to name to ip resolve failed for ip address: *.*.*.<xx1> (!= )
bdachecknet: ip to name to ip resolve failed for ip address: *.*.*.<xx2> (!= )
bdachecknet: ip to name to ip resolve failed for ip address: *.*.*.<xx3> (!= )
bdachecknet: ip to name to ip resolve failed for ip address: *.*.*.<xx4> (!= )
bdachecknet: ip to name to ip resolve failed for ip address: *.*.*.<xx5> (!= )

In this environment "imageinfo" reports a "May 09" timestamp:

# imageinfo

Big Data Appliance Image Info

IMAGE_CREATION_DATE : Mon May 09 15:35:07 EDT 2016
IMAGE_LABEL : BDA_4.5_LINUX.X64_RELEASE
IMAGE_VERSION : 4.5.0
LINUX_VERSION : Oracle Linux Server release 6.7
KERNEL_VERSION : 2.6.39-400.264.1.el6uek.x86_64
BDA_RPM_VERSION : bda-4.5.0-1.el6.x86_64
OFED_VERSION : OFED-IOV-1.5.5-2.0.0088
JDK_VERSION : jdk1.8.0_92-1.8.0_92-fcs.x86_64
HADOOP_VERSION : 2.6.0-cdh5.7.0

The expected/latest timestamp on "imageinfo" should be "May 18":

# imageinfo

Big Data Appliance Image Info

IMAGE_CREATION_DATE : Wed May 18 12:48:55 PDT 2016
IMAGE_LABEL : BDA_4.5_LINUX.X64_RELEASE
IMAGE_VERSION : 4.5.0
LINUX_VERSION : Oracle Linux Server release 6.7
KERNEL_VERSION : 2.6.39-400.264.1.el6uek.x86_64
BDA_RPM_VERSION : bda-4.5.0-1.el6.x86_64
OFED_VERSION : OFED-IOV-1.5.5-2.0.0088
JDK_VERSION : jdk1.8.0_92-1.8.0_92-fcs.x86_64
HADOOP_VERSION : 2.6.0-cdh5.7.0

Cause

The problem cause is that the patch:

Patch 23241894: BDA 4.5.0 PACKAGE FOR CONFIGURING NEW BDA RACKS WITH BASE IMAGE LESS THAN V4.5.0

has not been applied on this system.

The main intention of Patch 23241894 is to allow the additional network configuration functionality in BDA v4.5.0 to be used with BDAs with a lower v4.x Base Image. However, this patch also fixes a network configuration bug that is present in the bda rpm version shipped with BDA Mammoth v4.5.0. This is the bug that causes bdachecknet to fail.

The bug is not present in the bda v4.5.0 rpm in the patch nor is the bug present in the bda v4.5.0 rpm in the BDA 4.5.0 base image on ARU (or shipped from the factory). Hence this bug can be encountered on systems upgraded to BDA Mammoth v4.5.0. In such cases the patch can be applied to resolve the issue with bdachecknet.

Note: This patched bda 4.5.0 rpm was released just after BDA v4.5.0 Mammoth was released. It is also the case that the bda package version was not incremented in the patched rpm so the different code versions (original and patched) have the same version number. This should not have happened.

Solution

To workaround the issue, follow the steps below. Perform all steps on Node 1 of the cluster as 'root' user unless otherwise specified.

Please note there is no downtime required to install this patch.

1. Ensure you are on a BDA V4.5 system without Patch 23241894. Verify this with "imageinfo". Note the date "May 09" indicates Patch 23241894 is not applied.

# imageinfo

Big Data Appliance Image Info

2. Also verify that you do not have the latest bda rpm for BDA v4.5 with "rpm -qi bda-4.5.0-1.el6.x86_64". Note the "Mon 09 May 2016" build date indicates the latest bda rpm is not on the system.

# dcli -C "rpm -qi bda-4.5.0-1.el6.x86_64"

Output is like:

# dcli -C "rpm -qi bda-4.5.0-1.el6.x86_64"

*.*.*.1: Name : bda Relocations: /opt/oracle/bda
*.*.*.1: Version : 4.5.0 Vendor: (none)
*.*.*.1: Release : 1.el6 Build Date: Mon 09 May 2016 12:35:07 PM PDT
*.*.*.1: Install Date: (not installed) Build Host: adc00owq.us.oracle.com
*.*.*.1: Group : BDA Software Development Source RPM: bda-4.5.0-1.el6.src.rpm
*.*.*.1: Size : 160663451 License: Oracle Inc.
*.*.*.1: Signature : (none)
*.*.*.1: Summary : Oracle Engineered Systems bda software
*.*.*.1: Description :
*.*.*.1: Oracle Big Data Appliance support files and programs.
*.*.*.1: Original build location : BDA_4.5_LINUX.X64_RELEASE
...

3. Confirm the cluster is healthy:

a) Confirm cluster is healthy in CM.
b) Run bdacheckcluster and confirm it is successful
c) Run the cluster verification checks on Node 1 of the cluster as 'root'. Ensure they are successful.

# cd /opt/oracle/BDAMammoth
# ./mammoth -c

4. From My Oracle Support download Patch 23241894: BDA 4.5.0 PACKAGE FOR CONFIGURING NEW BDA RACKS WITH BASE IMAGE LESS THAN V4.5.0

5. Place the patch in a staging directory on Node 1 of the cluster, for example at: /home/stage_bda and copy the patch to all nodes of the cluster.

For example:

a) Verify the patch in /home/stage_bda:

# cd /home/stage_bda

# ls -l

total 121484
-rw-r--r-- 1 root root 124399049 Nov 4 06:31 p23241894_450_Linux-x86-64.zip

# cksum *

m2148112979 124399049 p23241894_450_Linux-x86-64.zip

# md5sum *

f5e8d3b5b4a5665d37310cb65504682f p23241894_450_Linux-x86-64.zip

b) Uncompress the patch: p23241894_450_Linux-x86-64.zip:

# unzip p23241894_450_Linux-x86-64.zip

Archive: p23241894_450_Linux-x86-64.zip
creating: bda-4.5.0-package/
inflating: bda-4.5.0-package/README.txt
inflating: bda-4.5.0-package/bda-4.5.0-1.el6.x86_64.rpm

c) Copy the file bda-4.5.0-package/bda-4.5.0-1.el6.x86_64.rpm to all nodes of the cluster

For example:

# dcli -C mkdir /home/stage_bda
# dcli -C -f /home/stage_bda/bda-4.5.0-package/bda-4.5.0-1.el6.x86_64.rpm -d /home/stage_bda

d) Verify:

# dcli -C ls -ltr /home/stage_bda/bda-4.5.0-1.el6.x86_64.rpm

*.*.*.1: -rw-r--r-- 1 root root 124495920 Nov 4 06:41 bda-4.5.0-1.el6.x86_64.rpm
...
*.*.*.n: -rw-r--r-- 1 root root 124495920 Nov 4 06:41 bda-4.5.0-1.el6.x86_64.rpm

6. On each node individually, NOT with dcli:

a) Remove the existing package first. This is necessary because the version number is the same on the original and patch bda rpms.

# rpm -e bda-4.5.0-1.el6.x86_64

b) Run the upgrade command "rpm -Uvh /path/bda-4.5.0-1.el6.x86_64.rpm":

For example:

# rpm -Uvh /home/stage_bda/bda-4.5.0-1.el6.x86_64.rpm

Output is like this on each node:

# rpm -e bda-4.5.0-1.el6.x86_64
# rpm -Uvh /home/stage_bda/bda-4.5.0-1.el6.x86_64.rpm

Preparing... ########################################### [100%]
1:bda ########################################### [100%]

7. Verify the new bda rpm is installed. Note the May 18 Build date.

# dcli -C "rpm -qi bda-4.5.0-1.el6.x86_64"

*.*.*.1: Name : bda Relocations: /opt/oracle/bda
*.*.*.1: Version : 4.5.0 Vendor: (none)
*.*.*.1: Release : 1.el6 Build Date: Wed 18 May 2016 12:48:55 PM PDT
*.*.*.1: Install Date: Fri 04 Nov 2016 06:58:30 AM PDT Build Host: slc09tgn.us.oracle.com
*.*.*.1: Group : BDA Software Development Source RPM: bda-4.5.0-1.el6.src.rpm
*.*.*.1: Size : 160663513 License: Oracle Inc.
*.*.*.1: Signature : (none)
*.*.*.1: Summary : Oracle Engineered Systems bda software
*.*.*.1: Description :
*.*.*.1: Oracle Big Data Appliance support files and programs.
*.*.*.1: Original build location : BDA_4.5_LINUX.X64_RELEASE
...

8. Confirm the cluster is healthy:

a) Confirm cluster is healthy in CM.
b) Run bdacheckcluster and confirm it is successful
c) Run the cluster verification checks on Node 1 of the cluster as 'root'. Ensure the output is sucessful.

# cd /opt/oracle/BDAMammoth
# ./mammoth -c

9. Run bdachecknet and verify the admin test now passes.

Attachments

This solution has no attachment