How to Reimage an Oracle Big Data Appliance When the Entire Network Structure/IP Definitions Will Change

Asset ID:	1-79-1553330.1
Update Date:	2017-11-01
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1553330.1 : How to Reimage an Oracle Big Data Appliance When the Entire Network Structure/IP Definitions Will Change

Applies to:

Big Data Appliance Hardware
Big Data Appliance X3-2 Hardware
Big Data Appliance Integrated Software - Version 1.0.3 and later
Big Data Appliance X5-2 Hardware
Linux x86-64

Purpose

This document provides the steps for reimaging an Oracle Big Data Appliance (BDA) when the entire network structure/IP definitions will change. This scenario can arise when a machine is being moved from one site to another site. One example is the case of an Oracle Big Data Appliance (BDA) moving from one customer site to another customer site for a Proof of Concept project.

Details

The general steps for reimaging a BDA rack moving from one site to another where the entire network structure/IP definitions will be different are presented here.

Note: In the case where network IP definitions have changed a reimage of all cluster nodes is absolutely required. The same holds true if a cluster previously existed on the rack.

At site One i.e. the site where the BDA is transferring away from:

1. Disconnect the rack from the network on site one. Disconnect both the:

a) Administrative network. The administrative network is the 1 gigabit Ethernet (GbE) network that connects to the site's existing administrative network and is used to administer all components of Oracle Big Data Appliance.

b) Client Access network. The client access network is the 10 GbE network which connects the servers though the gateway switches to the site's existing client network and is used for client access to the servers

2. Re-image the BDA rack to factory settings via the steps below. Follow these steps and do not directly re-image to the new IP settings.

a) Download the latest (or required) BDA Base Image tarball from My Oracle Support.

b) Copy the BDA image tarball onto the first (bottom) server of the BDA to be reimaged.

c) As root unzip the base image.

d) To reimage the rack from existing customer network settings to the original factory network settings:

./reimagerack deploy ship

Detailed instructions for downloading the base image and other steps are available in the Oracle Big Data Appliance Base Image Version documents listed in the Patchset Master note at: Oracle Big Data Appliance Patch Set Master Note (Doc ID 1485745.1). For example the document for BDA V2.5 is:

Oracle Big Data Appliance Base Image Version 2.5.0 for New Installations on OEL6 (Doc ID 1640105.1)

Note that it is important to re-image at the original site one as this will also erase any proprietary data on the BDA.

Also note that if the reimage did not happen on the original site for some reason, this can be done at the destination site. However to reimage to factory settings at the destination site you must make sure the rack is not connected to the data center. See: Oracle Big Data Appliance Installation Frequently Asked Questions (FAQ) for details on what it means for the rack to be disconnected from the data center.

At site Two i.e. the site where the BDA is transferring to (Destination site):

3. Using the Oracle Big Data Appliance Configuration Utility generate new configuration files for the new site.

The details can be found in the 'Using the Oracle Big Data Appliance Configuration Utility' section of the Oracle® Big Data Appliance Owner's Guide Release and in the "Preinstall Instructions" section of the Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release MOS documents. You can locate all documents from the: Patchset Master note at: Oracle Big Data Appliance Patch Set Master Note (Doc ID 1485745.1). For example see the Preinstall Instructions in the the document for BDA V3.0.1 at:

Installing/Extending Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release V3.0.1 for New Installations with OL6 Base Image (Doc ID 1679743.1)

4. Install and configure the hardware for the Oracle Big Data Appliance at the new site.

Full details are in the, "Installing Oracle Big Data Appliance at the Site" section of the Oracle® Big Data Appliance Owner's Guide Release.

5. Follow the standard hardware installation process to configure the Cisco Ethernet Switch, Infiniband Leaf and Spine switches, PDUs and BDA servers to the new site's network settings. Specifically do this by following the steps in sections 7.2 to 7.5 as described in the 'Configuring Oracle Big Data Appliance' chapter of the Oracle® Big Data Appliance Owner's Guide Release. Specifically sections 7.2 to 7.5 are:

Configuring the Cisco Ethernet Switch
Configuring the InfiniBand Leaf and Spine Switches
Configuring the Power Distribution Units
Configuring the Oracle Big Data Appliance Servers

6. Make sure the following network configurations are in place:

Note that if you can not do this step until after networksetup-one (due to not being on the network) it is allowed to delay step 6 until after networksetup-one when the rack is connected to the network. But step 6 must be done before networksetup-two.

a) Make sure the nameservers as configured in /etc/resolv.conf on the leaf switches are accessible.

Do this as follows for each leaf switch:

(i) From the leaf switch ping the name server which is the first entry of /etc/resolv.conf.

# ping <name_server>

(ii) From a host (which is not a leaf switch), perform a DNS lookup for some valid host name on the same administration network as the leaf switch you pinged in step (i) above. This could be the other leaf switch or one of the PDUs for example.

$ host <valid host name on same admin network>

b) In /etc/ssh/sshd_config on the leaf switches enable the MaxStartups parameter but assign it the value 20 instead of the default 10. (Only update /etc/ssh/sshd_config on the leaf switches. There is no need to do this on the spine switch.) On the leaf switches edit /etc/ssh/sshd_config to use:

MaxStartups 20

Instead of the default:

#MaxStartups 10

7. Follow the standard network configuration process as described in section 7.6 'Configuring the Network' of the 'Configuring Oracle Big Data Appliance' chapter of the Oracle® Big Data Appliance Owner's Guide Release. Note that this is the section where networksetup-one and networksetup-two are run.

8. Follow the standard Mammoth software deployment process. For details on the Mammoth release you are interested in see: Oracle Big Data Appliance Patch Set Master Note (Doc ID 1485745.1). For example for V3.0.1 see:

Installing/Extending Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release V3.0.1 for New Installations with OL6 Base Image (Doc ID 1679743.1)

Sample steps for BDA V3.0.1 installation when the BDA network structure/IP definitions changed

This example outlines the steps for installing BDA V3.0.1 when the network structure on the rack has changed. Following the steps in this note, some additional details for a V3.0.1 installation are:

1) Download Oracle Big Data Appliance Base Image Version 2.5.0 for New Installations on OEL6, Patch 18435972.

2) Use the downloaded Oracle Big Data Appliance Base Image Version 2.5.0 for Oracle Linux 6 to follow the instructions in this document: 1553330.1

a) These instructions have you run: './reimagerack deploy ship', to reimage the rack from existing customer network settings to original factory network settings.

# ./reimagerack deploy ship

Passwordless SSH set up to all hosts : <admin_IP1> <admin_IP2> ... <admin_IPn>

Using /var/tmp/BDABaseImage-ol6-2.5.0_RELEASE/BDABaseImage-ol6-2.5.0_RELEASE.iso to re-image
BDABaseImage-ol6-2.5.0_RELEASE.iso: OK

FINAL CONFIRMATION : All of the following hosts will be completely erased, re-imaged and initialized
including resetting of the iloms: <admin_IP1> <admin_IP2> ... <admin_IPn>

Proceed [y/n]? y

Running hardware checks on all nodes

Calling makebdaimage to generate a rack network configuration file
makebdaimage: -usbrack /var/tmp/BDABaseImage-ol6-2.5.0_RELEASE/BDABaseImage-ol6-2.5.0_RELEASE.iso /opt/oracle/bda/BdaDeploy.json /opt/oracle/bda/BdaShip.json
BDABaseImage-ol6-2.5.0_RELEASE.iso: OK
makebdaimage: fetching macs
`/tmp/server-BDABaseImage/bda-reimage.json' -> `/tmp/bda-reimage-rack.json'

Completed generating network configuration file for rack re-image

Copying makebdaimage to all servers

Copying /var/tmp/BDABaseImage-ol6-2.5.0_RELEASE/BDABaseImage-ol6-2.5.0_RELEASE.iso to all servers

Copying biosconfig to all servers

Copying ubiosconfig to all servers

Copying rack network configuration file to all servers

Preparing internal USB disk for re-imaging on all servers

Rebooting all servers

Broadcast message from root@bdanode01.example.com
(unknown) at 13:21 ...

The system is going down for reboot NOW!
# Connection to <admin_IP1> closed by remote host.
Connection to <admin_IP1> closed.

Note: When './reimagerack deploy ship' is complete, as in the case of running './reimagerack' the first reboot reboots into the reimaging process. After that there will be a /root/BDA_IMAGING_SUCCEEDED on all servers. A second reboot will be necessary before deploying Mammoth. After the second reboot there will be a file /root/BDA_REBOOT_SUCCEEDED on all servers. This is required before proceeding.

b) Download the latest, Oracle BDA Configuration Generation Utility 3.0.1 (supports Cloudera CDH 5.0.1) and use it to generate the new BdaDeploy.json, and cluster_name-config.json and other configuration files for V3.0.1.

Note: The V3.0.1 BdaDeploy.json is backward compatible to BDA V2.3.1.

c) Continue on with the rest of the configuration steps in this document: Doc ID 1553330.1.

d) Using the V3.0.1 generated BdaDeploy.json, and cluster_name-config.json, do the Mammoth install for V3.0.1. Follow the steps in: Installing/Extending Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release V3.0.1 for New Installations with OL6 Base Image (Doc ID 1679743.1)

Note: Make sure to run the generated 'bda-preinstall-checkip.sh' as per the documentation section '4.4 Validating the Network Settings' in the Oracle Big Data Appliance Owner's Guide Release 3 (3.0).

Note: After running ./BDAMammoth-ol6-3.0.1.run, copy over the V3.0.1 generated cluster_name-config.json and continue on with the Mammoth V3.0.1 installation.

To summarize you go directly from './reimagerack deploy ship' with V2.5 to installing V3.0.1 with the newly generated files, plus the additional configuration steps in the note.

Attachments

This solution has no attachment