Oracle Database Appliance X4-2
 
Oracle Database Appliance Software
 
Oracle Database Appliance X3-2
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Database Appliance>DB: ODA_EST
 

The ODA (Oracle Database Appliance) X4-2 and X3-2 have the option of an additional second Storage Disk Array (DE 24p) doubling the storage capacity. Adding this seconds JBOD is designed to be as simple: Racking, proper Cabling, and powering up the unit. This ODA then detects ths Second JBOD (Just a bunch of Disks) and automatically configures ASM storage. Under most circumstance this second JBOD can be added without user intervention other than cabling and powering up the unit. However, if the ODA is not updated on a regular (Quarterly) basis there is potential for a gap between the installed ODA and newer JBOD. This  gap between older ODAs and newer manufactured JBOD hardware and firmware is expected. This note will discuss some of the situations that may be encountered adding a second JBOD including strategies to avoid problems due to older and newer hardware configurations. A discussion will follow to help provide best practices and tips for future usage.

Applies to:

Purpose

After the release of the original Oracle Database Appliance (ODA)v1, the newer hardware versions have the option of an additional second Storage Disk Array. For the X4-2 and X3-2 the second shelf are identical (DE 24p) doubling the storage capacity. For the X5-2 a unique shelf is added which includes a different configuration and size of disks ( not swappable with X3-2 or X4-2 storage). This Second JBOD (Just a Bunch Of Disks) automatically configures ASM storage. Under most circumstance this second JBOD can be added without user intervention other than cabling and powering up the unit. For various reasons JBOD addition is not always successful. . This Note has four parts. 1) Proactive - Best Practice before adding a JBOD.  2) Reactive which will discuss some of the situations that may be encountered adding a second JBOD. For example, due to natural occurring updates newer JBOD disks and firmware may result in a gap between older ODA configuration and newer manufactured JBODs

Troubleshooting Steps

SUMMARY of PROACTIVE STEPS

1. Do all ODA patching up to date prior to JBOD2 addition - Confirm Completed*

2. Confirm both nodes are at the same version and patchset level

3. Check the Disk Health on the existing JBOD

4. Confirm ASM,  and CRS health on both Nodes -  if already on 12.x add ACFS checks
   

5. Confirm SSH communications using shared passwords works for all users and between nodes
   

6. Confirm space is available - A minimum of 10gb per node for /u01
   

7. Make sure the OAKD is available and running on both nodes 

Adding the JBOD

1. Cable the storage expansion shelf according to the ODA setup poster*

2. Login to each server node and run oakcli to validate the storage cabling

3. Power On the Second JBOD and confirm the JBOD is recognized and added

4. Confirm the JBOD2 disks have been added to the OS

5. Confirm the JBOD2 disks have been added to ASM
   

ODM IC , IV and DC

SUMMARY of REACTIVE STEPS

PROACTIVE CHECKS

Do all patching prior to JBOD addition

 * Check your version to Confirm that a fix is in place for
      Bug 23244767 - "OAKCLI ADD DISK" OVERWRITES SLOT METADATA ON OPPOSITE ENCLOSURE E0/E1 W/ 2 JBOD
      - Fixed->12.1.2.8 ; Manual fix (script) available for 12.1.2.7 and 12.1.2.6

Confirm both nodes are at the same version and patchset level

Check the Disk Health of the existing JBOD

Confirm ASM and CRS health on both Nodes

 First node: Node0 

 More detail at the disk level 

Confirm communications between both nodes: Confirm that SSH is working using the same password for oracle, root and grid.

From node0:

Confirm there is enough space available on each node - At a minimum of 10gb

Confirm that OAKD is running on both nodes

There are several commands to confirm this including

From each node

If oakd is not running on either node do not continue adding the JBOD until this problem is fixed!

Preparing to Add the JBOD

 Cable the storage expansion shelf according to the ODA setup poster*

 Login to each server node and run oakcli to validate the pre-JBOD2 ODA storage cabling

The correct results will confirm if the two ODA Server Nodes are properly cabled to the Original Storage Shelf  

All Current DISKS for the original JBOD should be confirmed as online, good and added to the existing Diskgroups on both nodes.

Check before powering on JBOD2

Both Nodes plus the original JBOD are up and running when adding the second JBOD

Power On the Second JBOD and confirm the new JBOD is recognized 

You can monitor activity directly from the JBOD
However the best Indicator that the second JBOD is being recognized is using the oakcli validate -c storage topology  command again:

Make sure that there are no failures reported before proceeding

Confirm the JBOD2 disks have been added to the OS

Confirm the JBOD2 disks have been added to ASM

 
Example 

REACTIVE

1) Make sure that the disks are recognized at the OS level 

   Check

Confirm the disk groups for All ODA HW types:

Note that ODA X5-2 also adds a FLASH diskgroup 

2) If the OS and topology are correct but the disks are not added to ASM use the following steps

COMMENT: Until the disks are recognized as GOOD and ONLINE do not attempt the following

New disks show as mpath vs actual name or are not added back

5. oakcli add jbod disk on both nodes

ONLY use the following commands IF the disk are not showing as added in the ASM ALERT.LOGs

6. The following command should only be used when all _Original_ JBOD disks are confirmed as healthy and online

   oakcli xxxxxxx

NOTE: DO NOT use the above command if ASM redundancy is broken due to offlined disks

For pre-12.1.2.2 we suggest using manual commands to add the disks

NOTE: if the disks are not recognized at the OS then you are at a stopping point
          If ONLY the SSDs are not recognized then you need to download and add the new validation_props.XML from this note and add to both nodes, flush multipath information,

You can add the second JBOD diskgroup disks using sql scripts without errors and other problems no longer occurred
You can add disks to ASM using three scripts for the 3 diskgroups

Manually extract the disk information and name from current asmappl.config for E1 (which is the second / new JBOD)
Two parts: the path for '/dev/mapper/...diskname...'   and NAME   '...diskname...' 

 
Manually adding disks to the diskgroups

1) Alter diskgroup ADD DISK to RECO ( p2 )  for the new JBOD E1 -- 20 HDDs

Example for RECO

2) Alter diskgroup ADD DISK to REDO ( p1 )  for the new JBOD E1 -- four SSDs*

 * adding the REDO diskGROUP may be necessary if 2 or more REDO disks are offline

3) Alter diskgroup add disk to DATA ( p1 )  for the new JBOD E1 -- 20 HDDs

EXAMPLE ONLY !!!

alter diskgroup reco add disk
'/dev/mapper/HDD_E1_S00_1468978072p2' name HDD_E1_S00_1468978072p2,
'/dev/mapper/HDD_E1_S01_1468972312p2' name HDD_E1_S01_1468972312p2,
'/dev/mapper/HDD_E1_S02_1468871052p2' name HDD_E1_S02_1468871052p2,
'/dev/mapper/HDD_E1_S03_1468783288p2' name HDD_E1_S03_1468783288p2,
'/dev/mapper/HDD_E1_S04_1468970364p2' name HDD_E1_S04_1468970364p2,
'/dev/mapper/HDD_E1_S05_1468970612p2' name HDD_E1_S05_1468970612p2,
'/dev/mapper/HDD_E1_S06_1468992196p2' name HDD_E1_S06_1468992196p2,
'/dev/mapper/HDD_E1_S07_1468974660p2' name HDD_E1_S07_1468974660p2,
'/dev/mapper/HDD_E1_S08_1468964860p2' name HDD_E1_S08_1468964860p2,
'/dev/mapper/HDD_E1_S09_1468989020p2' name HDD_E1_S09_1468989020p2,
'/dev/mapper/HDD_E1_S10_1468955112p2' name HDD_E1_S10_1468955112p2,
'/dev/mapper/HDD_E1_S11_1468989280p2' name HDD_E1_S11_1468989280p2,
'/dev/mapper/HDD_E1_S12_1468870404p2' name HDD_E1_S12_1468870404p2,
'/dev/mapper/HDD_E1_S13_1468909960p2' name HDD_E1_S13_1468909960p2,
'/dev/mapper/HDD_E1_S14_1468971472p2' name HDD_E1_S14_1468971472p2,
'/dev/mapper/HDD_E1_S15_1468988588p2' name HDD_E1_S15_1468988588p2,
'/dev/mapper/HDD_E1_S16_1468835116p2' name HDD_E1_S16_1468835116p2,
'/dev/mapper/HDD_E1_S17_1468964292p2' name HDD_E1_S17_1468964292p2,
'/dev/mapper/HDD_E1_S18_1468919684p2' name HDD_E1_S18_1468919684p2,
'/dev/mapper/HDD_E1_S19_1468919780p2' name HDD_E1_S19_1468919780p2;

EXAMPLE ONLY !!!

alter diskgroup redo add disk 
'/dev/mapper/SSD_E1_S20_1309260236p1' name SSD_E1_S20_1309260236p1,
'/dev/mapper/SSD_E1_S21_1309257240p1' name SSD_E1_S21_1309257240p1,
'/dev/mapper/SSD_E1_S22_1309309524p1' name SSD_E1_S22_1309309524p1,
'/dev/mapper/SSD_E1_S23_1309256372p1' name SSD_E1_S23_1309256372p1;

EXAMPLE ONLY !!!

alter diskgroup data add disk
'/dev/mapper/HDD_E1_S00_1468978072p1' name HDD_E1_S00_1468978072p1,
'/dev/mapper/HDD_E1_S01_1468972312p1' name HDD_E1_S01_1468972312p1,
'/dev/mapper/HDD_E1_S02_1468871052p1' name HDD_E1_S02_1468871052p1,
'/dev/mapper/HDD_E1_S03_1468783288p1' name HDD_E1_S03_1468783288p1,
'/dev/mapper/HDD_E1_S04_1468970364p1' name HDD_E1_S04_1468970364p1,
'/dev/mapper/HDD_E1_S05_1468970612p1' name HDD_E1_S05_1468970612p1,
'/dev/mapper/HDD_E1_S06_1468992196p1' name HDD_E1_S06_1468992196p1,
'/dev/mapper/HDD_E1_S07_1468974660p1' name HDD_E1_S07_1468974660p1,
'/dev/mapper/HDD_E1_S08_1468964860p1' name HDD_E1_S08_1468964860p1,
'/dev/mapper/HDD_E1_S09_1468989020p1' name HDD_E1_S09_1468989020p1,
'/dev/mapper/HDD_E1_S10_1468955112p1' name HDD_E1_S10_1468955112p1,
'/dev/mapper/HDD_E1_S11_1468989280p1' name HDD_E1_S11_1468989280p1,
'/dev/mapper/HDD_E1_S12_1468870404p1' name HDD_E1_S12_1468870404p1,
'/dev/mapper/HDD_E1_S13_1468909960p1' name HDD_E1_S13_1468909960p1,
'/dev/mapper/HDD_E1_S14_1468971472p1' name HDD_E1_S14_1468971472p1,
'/dev/mapper/HDD_E1_S15_1468988588p1' name HDD_E1_S15_1468988588p1,
'/dev/mapper/HDD_E1_S16_1468835116p1' name HDD_E1_S16_1468835116p1,
'/dev/mapper/HDD_E1_S17_1468964292p1' name HDD_E1_S17_1468964292p1,
'/dev/mapper/HDD_E1_S18_1468919684p1' name HDD_E1_S18_1468919684p1,
'/dev/mapper/HDD_E1_S19_1468919780p1' name HDD_E1_S19_1468919780p1;

ODM METHODOLOGY

ISSUE CLARIFICATION

JBOD level

DISK level   [A subset of disks] 

KEY INDICATORS and QUESTIONS to ASK

ISSUE VERIFICATION

IV is the results of answering questions in IC and supporting evidence from DC

DATA COLLECTION

KEY INFORMATION to COLLECT -- BASIC Configuration information 

Minimum Data Collection
  
Node0

Both Nodes

  Script JBODMinODA

Medium Data Collection  -- Use the Minimum Data Collection above Plus the following

>Script JBODmed

Full Data Collection

Medium +

> Script JBODfull

In progress

JBOD Troubleshooting QUESTIONAIRE:
After completing the Proactive Steps and you have a problem

1. Are you using 2.10 or lower?
   replace the validation_props.xml file on both nodes  
2.  Are you seeing GMON messages and the RECO diskgroup offline on one node?
   - Power on the second node and manually add the diskgroup RECO back to ASM
3.  ...

COMMENT: SUPPORT

TIPS

To set Node0 as the Master node

TIP:  To reduce the ASM Rebalance time   -- this can impact performance ongoing operations on other ODA if executed concurrent to load ( if no load try power levels from 32 up to 100, if light load try 11 up to 32)

TIP:

        Likewise, the addition of the second JBOD includes checks across both nodes. While not requiring a password change it is
        important to confirm that ssh does work across the nodes and all users can connect as expected using their shared password.

TIP

      To confirm that the second JBOD has been successfully detected is one step;
      to add disks to the OS is the second step;
      and to add all the disks to ASM diskgroups is the last step

     This command will confirm if you have successfully detected the new second JBOD.
     Once successfully added to the ODA systems this new storage shelf will be identified as JBOD1 in this validate command
     Also: All disks in the new JBOD1 will have a prefix of E1 for all physical disks.
     You will see the EBOD count go from 2 to 4, the JBOD name will change to JBOD0 and the second will be named JBOD1, and the JBOD count go from 1 to 2

     To confirm that disks have been added run "oakcli show disk"  -- It will show 48 entry for both jbod.
     E1 is the identified for the Second JBOD -- For the second JBOD to be added you must show e1_pd_xx resources.
     They must be in online good state before you issue   oakcli add JBOD disk    and then   oakcli add JBOD asm

     As the GRID user, you can connect to +ASM1 and +ASM2 Query v$asm_disk.
     You must see ASM disk HDD_E1_SXX_* and SSD_E1_SXX_*. to confirm that the new JBOD E1 disks have been added
     Mount_state and mode_state must be in cached and online state respectively. 
     It must show 88 entries for both jbods.

TIP:   For more information on ASM run the first three scripts from Note 470211.1

References

This solution has no attachment