| Asset ID: |
1-71-1901817.1 |
| Update Date: | 2018-04-20 |
| Keywords: | |
Solution Type
Technical Instruction Sure
Solution
1901817.1
:
How To Add a New Expansion Storage JBOD to an Oracle Database Appliance
| Related Items |
- Oracle Database Appliance X4-2
- Oracle Database Appliance X3-2
- Oracle Database Appliance X5-2
- Oracle Database Appliance Software
|
| Related Categories |
- PLA-Support>Sun Systems>x86>Engineered Systems HW>SN-x64: ORA-DATA-APP
|
This Note documents the steps to add an expansion storage to an existing Oracle Database Appliance X3-2,X4-2,X5-2,X6-2
In this Document
Applies to:
Oracle Database Appliance X4-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance Software - Version 2.6.0.0 to 2.10.0.0
Oracle Database Appliance X3-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Linux x86-64
Oracle Database Appliance running with Storage expansion
Goal
Steps to add an expansion storage shelf to an existing Oracle Database Appliance X3-2/X4-2/X5-2/X6-2
Solution
Before adding an expansion shelf, please make sure that you are running Software release 12.1.2.0 and above. The reason is that the new expansion jbods have HGST SSD's, which are only supported with Software release 12.1.2.0 and above. If customer's have an older software version, the SSDs will not get recognized and incorported into the appropriate ASM diskgroups. However, it is recommended that the system be at 12.1.2.12.0 for stability reasons.
Step 1: Cable the storage expansion shelf according to the setup poster.
The ODA X6-2 setup poster is here https://docs.oracle.com/cd/E89147_01/doc.122/e76904.pdf
The ODA X5-2 setup poster is here http://docs.oracle.com/cd/E22693_01/doc.12/e55694.pdf
The ODA X4-2 setup poster is here http://docs.oracle.com/cd/E22693_01/doc.12/e41668.pdf
The ODA X3-2 setup poster is here http://docs.oracle.com/cd/E22693_01/doc.12/e35554.pdf
The below diagram is from an ODA X3-2, but the cabling for the expansion shelf is exactly the same:
Step 2: Turn on the storage expansion shelf.
It is not possible to remotely control power to the storage, so you will need to turn on the power manually:
Step : Login to each server node and run oakcli to validate the storage cabling.
# oakcli validate -c storagetopology
It may take a while. Please wait...
INFO : ODA Topology Verification
INFO : Running on Node0
INFO : Check hardware type
SUCCESS : Type of hardware found : X3-2
INFO : Check for Environment(Bare Metal or Virtual Machine)
SUCCESS : Type of environment found : Virtual Machine(ODA BASE)
SUCCESS : Number of External LSI SAS controller found : 2
INFO : Check for Controllers correct PCIe slot address
SUCCESS : External LSI SAS controller 0 : 00:15.0
SUCCESS : External LSI SAS controller 1 : 00:16.0
INFO : Check if JBOD powered on
SUCCESS : 2JBOD : Powered-on
INFO : Check for correct number of EBODS(2 or 4)
SUCCESS : EBOD found : 4
INFO : Check for External Controller 0
SUCCESS : Cable check for port 0 on controller 0
SUCCESS : Cable check for port 1 on controller 0
SUCCESS : Overall Cable check for controller 0
INFO : Check for External Controller 1
SUCCESS : Cable check for port 0 on controller 1
SUCCESS : Cable check for port 1 on controller 1
SUCCESS : Overall Cable check for controller 1
INFO : Check for overall status of cable validation on Node0
SUCCESS : Overall Cable Validation on Node0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< make sure you get the same message for Node1
SUCCESS : JBOD0 Nickname set correctly : Oracle Database Appliance - E0
SUCCESS : JBOD1 Nickname set correctly : Oracle Database Appliance - E1
Make sure that there are no failures reported befoe proceeding.
Step 4: Run oakcli to make sure all disks in the expansion jbod are listed.
# oakcli show disk
NAME PATH TYPE STATE STATE_DETAILS
e0_pd_00 /dev/sda HDD ONLINE Good
e0_pd_01 /dev/sdb HDD ONLINE Good
e0_pd_02 /dev/sdbw HDD ONLINE Good
e0_pd_03 /dev/sdbx HDD ONLINE Good
e0_pd_04 /dev/sdby HDD ONLINE Good
e0_pd_05 /dev/sdbz HDD ONLINE Good
e0_pd_06 /dev/sdca HDD ONLINE Good
e0_pd_07 /dev/sdcb HDD ONLINE Good
e0_pd_08 /dev/sdcc HDD ONLINE Good
e0_pd_09 /dev/sdcd HDD ONLINE Good
e0_pd_10 /dev/sdce HDD ONLINE Good
e0_pd_11 /dev/sdcf HDD ONLINE Good
e0_pd_12 /dev/sdcg HDD ONLINE Good
e0_pd_13 /dev/sdch HDD ONLINE Good
e0_pd_14 /dev/sdci HDD ONLINE Good
e0_pd_15 /dev/sdcj HDD ONLINE Good
e0_pd_16 /dev/sdck HDD ONLINE Good
e0_pd_17 /dev/sdcl HDD ONLINE Good
e0_pd_18 /dev/sdcm HDD ONLINE Good
e0_pd_19 /dev/sdcn HDD ONLINE Good
e0_pd_20 /dev/sdco SSD ONLINE Good
e0_pd_21 /dev/sdcp SSD ONLINE Good
e0_pd_22 /dev/sdcq SSD ONLINE Good
e0_pd_23 /dev/sdcr SSD ONLINE Good
e1_pd_00 /dev/sdaw HDD ONLINE Good
e1_pd_01 /dev/sdax HDD ONLINE Good
e1_pd_02 /dev/sdaa HDD ONLINE Good
e1_pd_03 /dev/sdab HDD ONLINE Good
e1_pd_04 /dev/sdac HDD ONLINE Good
e1_pd_05 /dev/sdad HDD ONLINE Good
e1_pd_06 /dev/sdae HDD ONLINE Good
e1_pd_07 /dev/sdaf HDD ONLINE Good
e1_pd_08 /dev/sdag HDD ONLINE Good
e1_pd_09 /dev/sdah HDD ONLINE Good
e1_pd_10 /dev/sdai HDD ONLINE Good
e1_pd_11 /dev/sdaj HDD ONLINE Good
e1_pd_12 /dev/sdak HDD ONLINE Good
e1_pd_13 /dev/sdal HDD ONLINE Good
e1_pd_14 /dev/sdam HDD ONLINE Good
e1_pd_15 /dev/sdan HDD ONLINE Good
e1_pd_16 /dev/sdao HDD ONLINE Good
e1_pd_17 /dev/sdap HDD ONLINE Good
e1_pd_18 /dev/sdaq HDD ONLINE Good
e1_pd_19 /dev/sdar HDD ONLINE Good
e1_pd_20 /dev/sdas SSD ONLINE Good
e1_pd_21 /dev/sdat SSD ONLINE Good
e1_pd_22 /dev/sdau SSD ONLINE Good
e1_pd_23 /dev/sdav SSD ONLINE Good
Step 5. Run oakcli to check the health of components in expansion jbod.
# oakcli show enclosure
NAME SUBSYSTEM STATUS METRIC
E0_FAN0 Cooling OK 3220 rpm
E0_FAN1 Cooling OK 3000 rpm
E0_FAN2 Cooling OK 3750 rpm
E0_FAN3 Cooling OK 3000 rpm
E0_IOM0 Encl_Electronics OK -
E0_IOM1 Encl_Electronics OK -
E0_PSU0 Power_Supply OK -
E0_PSU1 Power_Supply OK -
E0_TEMP0 Amb_Temp OK 29 C
E0_TEMP1 Midplane_Temp OK 35 C
E0_TEMP2 PCM0_Inlet_Temp OK 38 C
E0_TEMP3 PCM0_Hotspot_Temp OK 46 C
E0_TEMP4 PCM1_Inlet_Temp OK 34 C
E0_TEMP5 PCM1_Hotspot_Temp OK 41 C
E0_TEMP6 IOM0_Temp OK 49 C
E0_TEMP7 IOM1_Temp OK 50 C
Step 6: Run oakcli to verify that all firmware components in the storage expansion are current.
# oakcli show version -detail
Reading the metadata. It takes a while...
System Version Component Name Installed Version Supported Version
-------------- --------------- ------------------ -----------------
2.10.0.0.0
Controller_INT 11.05.03.00 Up-to-date
Controller_EXT 11.05.03.00 Up-to-date
Expander 0018 Up-to-date
SSD_SHARED 9440 Up-to-date
HDD_LOCAL A606 Up-to-date
HDD_SHARED A600 Up-to-date
ILOM 3.1.2.10.d r83372 Up-to-date
BIOS 17050100 Up-to-date
IPMI 1.8.10.5 Up-to-date
HMP 2.2.6.5 Up-to-date
OAK 2.10.0.0.0 Up-to-date
OEL 5.9 Up-to-date
GI_HOME 11.2.0.4.0 Up-to-date
Step 7: If needed, perform firmware updates on the disk drives in the expansion jbod using oakcli.
The main components that you want to look at are the disks and expander in the expansion jbod. If any of these disks are less than the supported version, then you must perform firmware updates. Below is an example:
To upgrade the disk firmware, use the oakcli update -patch command, for example:
# oakcli update -patch 12.1.2.6.0 --infra <<<<<<<<<<<<<< be sure to use the version of your ODA software
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Both nodes may get rebooted automatically during the patch if required
Do you want to continue: [Y/N]?: y
INFO: User has confirmed the reboot
INFO: Patch bundle must be unpacked on the second node also before applying this patch
Did you unpack the patch bundle on the second node?: [Y/N]?: y
Please enter the 'root' user password:
Please re-enter the 'root' user password:
INFO: Setting up the SSH
...
...
For version 12.1.2.7.0 and above, the command syntax has changed to --storage:
# oakcli update -patch 12.1.2.7.0 --storage
After the infra patching has completed, run "oakcli show version -detail" again to make sure all disks are Up-to-date.
Step 8: check on status of the expansion shelf disks from an oakd and ASM perspective:
# oakcli show jbod status
INFO: 2015-09-30 21:56:52: Command provide status of expansion storage shelf disks in OAKD and ASM
INFO: 2015-09-30 21:56:52: Command output based on view when it's run. For monitoring online addition
INFO: 2015-09-30 21:56:52: of expansion storage shelf, run it few times to get correct picture. Online
INFO: 2015-09-30 21:56:52: addition of expansion storage shelf generally takes about 30 to 40 minutes
INFO: 2015-09-30 21:56:52: for all disks to be added to OAKD and ASM.
INFO: 2015-09-30 21:56:52: Present status of Expansion storage shelf
Disk Added Disk Not Added Status
OAKD 24 0 SUCCESS
ASM
DATA 15 1 SUCCESS
RECO 16 0 SUCCESS
REDO 4 0 SUCCESS
FLASH 4 0 SUCCESS
INFO: 2015-09-30 21:56:52: DiskGroup Mount Status
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB
3 RECO MOUNTED HIGH 69665408 36848980
2 FLASH MOUNTED NORMAL 3052416 383332
4 REDO MOUNTED HIGH 1526240 855932
1 DATA MOUNTED HIGH 50790400 26203284
If there are failures from the command above, refer to Doc ID 1959359.1: ODA Adding a Second JBOD : Proactive Steps, Data Collection Commands and Tips
This doc goes through the software steps of adding an additional storage shelf to an existing ODA system.
References
<NOTE:1959359.1> - ODA Adding a Second JBOD : Proactive Steps, Data Collection Commands and Tips
<NOTE:1561250.1> - JBOD Nickname is incorrectly set to : Oracle Database Appliance - ... E1
Attachments
This solution has no attachment
