How to Power Cycle an Oracle Big Data Appliance Node Using the ILOM when the Node is Not Reachable Using Ping/SSH

Asset ID:	1-79-1550440.1
Update Date:	2017-12-22
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1550440.1 : How to Power Cycle an Oracle Big Data Appliance Node Using the ILOM when the Node is Not Reachable Using Ping/SSH

Applies to:

Big Data Appliance X3-2 In-Rack Expansion - Version All Versions and later
Big Data Appliance Hardware - Version All Versions and later
Big Data Appliance X3-2 Hardware - Version All Versions and later
Big Data Appliance X3-2 Full Rack - Version All Versions and later
Big Data Appliance X3-2 Starter Rack - Version All Versions and later
Linux x86-64

Purpose

This document provides steps on how to power cycle an Oracle Big Data Appliance (BDA) node(which can not be contacted via ping nor can be accessed with ssh) using Oracle Integrated Lights Out Manager (ILOM) interface .

Details

The state of the node can be checked using the Oracle Integrated Lights Out Manager (ILOM) interface.

There are 2 options to connect to BDA node ILOM interface

a) ILOM CLI interface.

b) ILOM Web interface.

The ILOM web interface is the recommended option.

Note: There can be cases where node boot process is waiting for user input(due to disk being bad and clear cache needed , file system check errors etc ..). In such cases end user can enter the needed input thru ILOM Remote Console for the node to complete the boot scuessfully. Once the boot is sucessfull the node should be accessible thru SSH. So before doing a 'Power Cycle' would recommend to open a Remote Console session to find out the cause why the node is NOT reachable for reasons like being stuck in boot mode or maybe the node got booted from usb since OS disks are bad or network down .. etc .

Steps to Power Cycle BDA node thru ILOM Web interface

1. Go to Oracle ILOM Web GUI.
In a supported browser enter the IP Address or hostname for your Oracle Big Data Appliance server's ILOM. For list of browser's , Oracle ILOM supports refer to ILOM documentation .

The information about IP Address or hostname of ILOM's can be obtained from node 1 of the BDA in the file:

/opt/oracle/BDAMammoth/mammoth-<rack1name>.params

Look for a space separated list for:

ILOM_NAMES=(...)
ILOM_IPS=(...)

Note: If a security certificate error is displayed click on "Continue to this website (not recommended)." If a message "This Connection is Untrusted" is displayed then click on, "I Understand the Risks" and continue with the confirmations to log in.

2. Login as root. The default password is welcome1.

In ILOM Version 3.0.*
------------------------
3. Click the Remote Control tab (on the top right)

4. You can then click the Redirection tab and click on 'Launch Remote Console' to monitor the progress of the server shutdown/startup.

Launch Remote Console in 3.0

5. Click Remote Power Control tab and choose Power Cycle from the pull down list. Then click Save. The server will start up.

ILOM power cycle

In ILOM Version 3.1.*
------------------------

3. Expand 'Remote Control' in the left hand side menu and click on Redirection tab

4. Click Launch Remote Console to monitor the progress of the server shutdown/startup.

Launch Remote Console in 3.1

5. Expand 'Host Management' in the left hand side menu and click on 'Power Control'

6. Choose Power Cycle from the pull down list and then click Save. The server will start up.

Power Cycle in 3.1

Steps to Power Cycle BDA node thru ILOM CLI interface

1) SSH using IP Address or hostname for your Oracle Big Data Appliance server's ILOM. The default username/password is root/welcome1.

For example

# ssh root@bdanodename-ilom.us.oracle.com

Password:

Oracle(R) Integrated Lights Out Manager

Version 3.0.16.10.a r68533

Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.

2) 'reset /SYS' command can be used to power cycle a BDA node. Reset doesn't do a graceful shutdown(incase OS is up ) and 'stop /SYS' command can be used for a graceful shutdown. If using reset command then skip to step6 to monitor the reboot using serial console.

reset /SYS => This is power-off followed by power-on.

-> reset /SYS

Are you sure you want to reset /SYS (y/n)? y
Performing hard reset on /SYS

3) Use 'stop /SYS' command to power off the node which is NOT accessible via ping/ssh

-> help stop
The stop command is used to make a target unavailable.

Usage: stop [-script] [-force] [<target>]

Available options for this command:
-script : do not prompt for yes/no confirmation and act as if yes was specified
-force (-f) : specify that the action will be performed immediate

-> stop /SYS

Are you sure you want to stop /SYS (y/n)? y
Stopping /SYS

4) Check the power_state of the node using 'show /SYS' command

-> show /SYS

/SYS
    Targets:
       ....................

    Properties:
        type = Host System
        ipmi_name = /SYS
        product_name = SUN FIRE X4270 M2 SERVER
        product_part_number = 30057782+1+1
        product_serial_number = 1137FMM0C2
        product_manufacturer = ORACLE CORPORATION
        fault_state = OK
        clear_fault_action = (none)
        power_state = Off
....................................

5) Start the node using 'start /SYS' command

-> help start
The start command is used to make a target ready for use.

Usage: start [-script] [-force] [<target>]

Available options for this command:
-script : do not prompt for yes/no confirmation and act as if yes was specified
-force (-f) : specify that the action will be performed immediately

-> start /SYS

Are you sure you want to start /SYS (y/n)? y
Starting /SYS

6) To monitor the reboot process using Serial Console use 'start /SP/console' command.

During booting the serial console might NOT display since the default BIOS baud rate is 9600 but the serial setting is 115200. Set 'pendingspeed' to '9600' before starting the Serial Console.

-> cd /SP/serial/host

/SP/serial/host

-> set pendingspeed=9600

Set 'pendingspeed' to '9600'

-> set commitpending=true

Set 'commitpending' to 'true'

-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y

Serial console started. To stop, type ESC (

Once boot passes off to OS then display changes to junk/blank again due to baud-rate mis-match. Exit the serial console using 'ESC Shift 9' and change 'pendingspeed=115200' / commitpending=true, then connect back to the console using 'start /SP/console' command.

NOTE:- ILOM Remote Web/Serial console is basically a replication of the video monitor output, so one cannot cut and paste to it. Use "unalias ls" and it will take away the default colors to read the output clearly.

References

<NOTE:1550260.1> - Cloudera Manager Reports Bad or Concerning Health on Some Nodes

Attachments

This solution has no attachment