| Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback |
| Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition | ||
|
|
||
 |
||||||||||||||||||||||||||
Solution Type Technical Instruction Sure Solution 2097005.1 : ODA (Oracle Database Appliance): How To Detach/Attach a Previously Used JBOD and Move to Another ODA
In this Document
Applies to:Oracle Database Appliance - Version All Versions and laterOracle Database Appliance Software - Version 12.1.2.2 and later Information in this document applies to any platform. GoalScope of this article is describe how to detach a JBOD from one ODA and attach it later to another one. SolutionHow To Detach/Attach used JBOD to another ODARestrictions and important notes- If the second JBOD is currently used by the first ODA, in other words it's part of an ASM diskgroups (DATA, RECO, REDO, ...). You can't simply unplug second JBOD from one ODA, this will cause diskgroup to be dismounted because we lose majority of PST quorums - You can detach/attach a JBOD between the same ODA generation or between X3-2 and X4-2. You can not detach an X3-2/X4-2 JBOD and attach it to an X5-2 - Using the commands/steps described below on a system where the Second JBOD is added successfully may cause data loss. The StepsScenario 1 - JBOD attached and in use by a "source" ODASteps to be performed on source ODA(System from which JBOD is getting removed): 1. It's always best practice upgrade ODA to the latest software version before making these changes. Strongly recommended starting point version is 12.1.2.3.0 2. In case the JBOD is attached and in use by a "source" ODA, before detach it you need to perform a cleanup deploy executing: /opt/oracle/oak/onecmd/cleanupDeploy.pl
Note as this command will erase everything on such ODA, in particular the jbod header and it will reset the storage nickname.
Run following command to check JBOD nickname is unset: # fwr_exp_util list
You MUST NOT not see any JBOD with "Oracle Database Appliance - E0" or "Oracle Database Appliance - E1" as names, names would be set to "XXXXXXXXXXX"
Steps to be performed on Target ODA(System where JBOD going to attach). 1. Shutdown OAKD on both nodes. Ensure no oakd running on both nodes. oakcli stop oak
2. Connect 2nd jbod cables correctly to the "target" ODA. Once cables connected properly on both nodes, power on the 2nd jbod. After 2-3 minutes run storage topology tool on both nodes to verify cables connect are correct. oakcli validate -c StorageTopology
If you don't see message about jbod nicknames then check for following file # ls /opt/oracle/oak/conf/.topology
and if not present create it # touch /opt/oracle/oak/conf/.topology
rerun the "validate StorageTopology" command. Once storage topology is correct on both nodes proceed further. Otherwise recheck cables on system and recheck storage topology.
Don't proceed further unless both jbods are correctly cabled. 3. Once storage topology verified, we need to set nickname as 2nd jbod nickname was unset while clean deploy. Run following on node0 to set the nickname correctly. #/opt/oracle/oak/bin/stordiag/Topology_Validation.sh -deploy
Once done, rerun validate to make sure both jbod are nicknamed correctly. # oakcli validate -c storagetopology
Once storage topology is correct on both nodes proceed further. Otherwise recheck cables on system and recheck storage topology. Don't proceed further unless both jbods are correctly cabled.
4. Start oakd on both nodes one after another, keep a 1-2 minute gap between them. oakcli start oak
5. Once oakd on both nodes starts (wait 2-3 minuts), check disk resource status (on both nodes): example:
# oakcli show disk NAME PATH TYPE STATE STATE_DETAILS e0_pd_00 /dev/sda HDD ONLINE Good You must see all disk for 1st JBOD good and 2nd JBOD disk as NewDiskInserted. if you don't see above output on both nodes, don't proceed further.
6. Check for master/slave configuration using following command oakcli show ismaster
One node should be master and other slave. Make node 0 master and node1 as slave. To change role just restart oakd on master node then other node becomes master and this node become slave after oakd comes up. 7. Issue following command on both nodes. First on master node,then after 30-40 seconds on slave. oakcli add jbod disk
Above command will add 2nd jbod disks to oakd first. Till oak release 12.1.2.5, It takes about 20 to 25 minutes to add disks. You can check progress by query 'oakcli show disk | grep e1_pd_' from another terminal at 4-5 minutes intervals. From oak release 12.1.2.6 onward, above command returns in a minutes or so and disk will be added to oakd in background. So wait for 8-10 minutes after above command returns and then check disk status. You must see all disks resource for 2nd jbod should should be become online. Once all 24 disk added go to the next step, otherwise stop # oakcli show disk e0_pd_00 /dev/sda HDD ONLINE Good 8. Check that all /dev/mapper/ path for 2nd jbod has been populated and correct permission is set. You must have 20 p1 and p2 HDD partitions and 4 p1 SSD partitions on both nodes for X3-2 and X4-2: # ls -l /dev/mapper/HDD_E1_S*p2 | wc -l # ls -l /dev/mapper/SSD_E1_S*p1 | wc -l For X5-2: # ls -l /dev/mapper/HDD_E1_S*p2 | wc -l # ls -l /dev/mapper/SSD_E1_S*p1 | wc -l If all above condition satisfied proceed further otherwise stop.
9. Issue following command on both nodes. First on master node, then after 20-30 seconds on slave. oakcli add jbod asm
10. This will add 2nd jbod disks to ASM. Wait for 2-3 minutes then you check their status oakcli show jbod status
11. Once the disks are added to ASM, you may want to change the power limit at the ASM level to speed up the rebalance operation. Scenario 2 - JBOD was in use by a "source" ODA but actually detached from itFollowing steps needs to be run on Target System where 2nd JBOD going to be attached. 1. It's always best practice upgrade ODA to the latest software version before making these changes. Strongly recommended starting point version is 12.1.2.3.0 2. Shutdown OAKD on both nodes. Ensure no oakd running on both nodes. oakcli stop oak
3. Connect 2nd jbod cables correctly to the "target" ODA. Once cables connected properly on both nodes, power on the 2nd jbod. After 2-3 minutes run storage topology tool on both nodes to verify cables connect are correct. oakcli validate -c StorageTopology
If you don't see message about jbod nicknames then check for following file # ls /opt/oracle/oak/conf/.topology
and if not present create it # touch /opt/oracle/oak/conf/.topology
rerun the "validate StorageTopology" command. Don't proceed further unless both jbods are correctly cabled.
4. Once storage topology verified and If 2nd JBOD is not correctly nicknamed, run following on node0 to set the nickname correctly. #/opt/oracle/oak/bin/stordiag/Topology_Validation.sh -deploy
Once done, rerun validate to make sure both jbod are nicknamed correctly. # oakcli validate -c storagetopology
If you still see warning that JBOD nicknames are wrong, don't proceed further.
5. Start oakd on both nodes one after another, keep a 1-2 minute gap between them. oakcli start oak
6. Check the disk status # oakcli show disk NAME PATH TYPE STATE STATE_DETAILS e0_pd_00 /dev/sda HDD ONLINE Good you must see all disk for 1st JBOD good and 2nd JBOD disk as NewDiskInserted or OldDiskInserted(if oakd header on 2nd JBOD wasn't erased). You don't see above output on both nodes, don't proceed further.
7. Cleanup the attached JBOD issuing: oakcli erase jbod header
Such command is available on ODA version 12.1.2.2.0 and above. 8. Restart the OAKd oakcli restart oak
Wait for 2-3 minutes and check disk resource status # oakcli show disk e0_pd_00 /dev/sda HDD ONLINE Good you must see all disk for 1st JBOD good and 2nd JBOD disk as NewDiskInserted. You don't see above output on both nodes, don't proceed further.
9. Check for master/slave configuration using following command oakcli show ismaster
One node should be master and other slave. Make node 0 master and node1 as slave. To change role just restart oakd on master node then other node becomes master and this node become slave after oakd comes up. 10. Issue following command on both nodes. First on master node,then after 30-40 seconds on slave. oakcli add jbod disk
Above command will add 2nd jbod disks to oakd first. Till oak release 12.1.2.5, It takes about 20 to 25 minutes to add disks. You can check progress by query 'oakcli show disk | grep e1_pd_' resource after 4-5 minutes intervals. From oak release 12.1.2.6 onward, above command returns in a minutes or so and disk will be added to oakd in background. So wait for 8-10 minutes after above command returns and then check disk status. You must see all disks resource for 2nd jbod should should be become online. # oakcli show disk e0_pd_00 /dev/sda HDD ONLINE Good 11. Check that all /dev/mapper/ path for 2nd jbod has been populated and correct permission is set. You must have 20 p1 and p2 HDD partitions and 4 p1 SSD partitions on both nodes for X3-2 and X4-2: # ls -l /dev/mapper/HDD_E1_S*p2 | wc -l # ls -l /dev/mapper/SSD_E1_S*p1 | wc -l For X5-2: # ls -l /dev/mapper/HDD_E1_S*p2 | wc -l # ls -l /dev/mapper/SSD_E1_S*p1 | wc -l If all above condition satisfied proceed further otherwise stop.
12. Issue following command on both nodes. First on master node, then after 20-30 seconds on slave. oakcli add jbod asm
13. This will add 2nd jbod disks to ASM. Wait for 2-3 minutes then you check their status oakcli show jbod status
14. Once the disks are added to ASM, you may want to change the power limit at the ASM level to speed up the rebalance operation. References<BUG:23244767> - "OAKCLI ADD DISK" OVERWRITES SLOT METADATA ON OPPOSITE ENCLOSURE E0/E1 W/ 2 JBOD<NOTE:2134317.1> - ODA (Oracle Database Appliance) : How to fix wrong JBOD nickname <NOTE:1901817.1> - How To Add an Expansion Storage JBOD to an Oracle Database Appliance Attachments This solution has no attachment |
||||||||||||||||||||||||||
|
||||||||||||||||||||||||||