Things to Check in ASM When Replacing an ONLINE disk from Exadata Storage Cell

Asset ID:	1-71-1326611.1
Update Date:	2016-12-23
Keywords:

Solution Type Technical Instruction Sure

Solution 1326611.1 : Things to Check in ASM When Replacing an ONLINE disk from Exadata Storage Cell

Applies to:

Oracle Exadata Storage Server Software - Version 11.1.0.3.0 and later
Exadata Database Machine X2-2 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on x86-64 (64-bit)
Linux x86-64

Goal

Replace ONLINE disk(s) in Exadata storage cell due to various reason (such as increasing Media Error Count). An ONLINE disk for the purpose of this document is one which has not been dropped from the ASM diskgroups by the Exadata auto disk management features.

Solution

11.2.3.3.0 and later

For Exadata Storage Server Software versions 11.2.3.3.0 and later, a new cellcli command has been introduced to facilitate the process, "ALTER PHYSICALDISK <disk number> DROP FOR REPLACEMENT". Example:

CellCLI> ALTER PHYSICALDISK 20:2 DROP FOR REPLACEMENT

The command will perform checks to determine whether it is safe to remove the disk, then prepares it for replacement. Upon it's successful completion, the blue LED on the disk will have been turned on. A rough overview of the checks and actions that the command performs in the background:

- Checks ASM to see if the partner disks associated to the disk to be replaced are online. These partner disks reside in different storage cells. The command will fail if any of the partner disks are offline in ASM. This is equivalent to running "list griddisk attributes name, asmdeactivationoutcome"

- If ran against one of the two disks that the cell's operating system resides on (slots 0 and 1), it will check if the MD partner disks are in sync. If they are not, the command will fail

- If above checks are successful, the Management Server (MS) will request that the griddisks residing on the physicaldisk be marked as Inactive. This results in the griddisks being offlined in ASM

- The celldisk is dropped (from memory only) and the blue LED is turned on. The disk is now ready for physical replacement

Note that the command does not initiate a drop of the griddisks from ASM, instead opting for an offline of the disks. As such, a rebalance will not occur as a result of this command being run. The advantage to this approach is that the disk can be replaced as soon as the command completes and that a single rebalance operation will occur, the one that takes place after the new disk has been inserted. The approach relies on the integrity of the secondary (mirror) extents residing on the partner disks in order to restore redundancy after the disk is replaced. It is in many ways similar to what takes place after a disk fails. If other disks residing in different cells have or are experiencing I/O issues (such as increased Media Error Count or I/O errors in ASM or database instances alert logs) and they are still online in ASM and part of the diskgroups while this procedure is undertaken, it may be more prudent to instead follow the manual procedure below, which will result in disk being dropped from ASM and rebalance taking place. The rebalance would be able to access (bar any medium errors, bad blocks, etc) both the primary extents stored on the disk being dropped and the mirrored extents stored on the partner disks in order to restore full redundancy after the disk is dropped. This is in contrast to a single rebalance following new disk being inserted, which could only access the mirrored extents on the partner disks.

If after running "ALTER PHYSICALDISK DROP FOR REPLACEMENT", a decision is made to not physically replace the disk, or if the command was run against the incorrect disk, the following command can be run to re-enable the disk and re-add it back:

CellCLI> ALTER PHYSICALDISK 20:2 REENABLE

If a disk was removed and is re-added to a different slot, "cellcli -e list lun" may report a status of "normal - rejected due to wrong slot". To resolve, run:

CellCLI> ALTER PHYSICALDISK 20:2 REENABLE FORCE

Manual procedure (11.2.3.2.1 and earlier)

Steps 1 through 7 should be ideally performed several hours prior to the Field Engineer's scheduled arrival on site, as to allow the rebalance following the dropping of the disk from ASM to complete. While allowing the rebalance that commences after the disk is dropped to complete is not a requirement and may not be applicable to all failure scenarios, it may help avoid rare cases that can affect data integrity.

1. You have identified a disk to be replaced:

See: Determining when Disks should be replaced on Oracle Exadata Database Machine <Document 1452325.1>

2. You must make sure that no other disks that are partners to the disk to be replaced are offline or missing. There are up to 8 partner disks and are located in other cells (failgroups). NB:

If there are other disks offline on other storage cells and they happen to be partners of the disk being replaced, replacing the disk may result in all the diskgroups to be dismounted, database instances to crash and potential data loss. At any point of time, disks from two different storage cells should not be simultaneusly removed. Furthermore, no disk should be removed from a given storage cell (prior to performing the below checks) if one or more disks are already offline on another storage cell.

a) Run the following queries in ASM. By default, each failgroup should contain 12 disks in the RECO and DATA diskgroups and 10 disks in DBFS_DG. The mode_status should be CACHED and mount_status ONLINE

col path format a40
col name format a30
col failgroup format a30
set linesize 200
select group_number, name from v$asm_diskgroup;
select inst_id,group_number,failgroup,mount_status,count(*) from gv$asm_disk
group by inst_id,group_number,failgroup,mount_status;
select inst_id,group_number,failgroup,mode_status,count(*) from gv$asm_disk
group by inst_id,group_number,failgroup,mode_status;

b) On the storage cell that the disk to be removed is located, run the following cellcli query. Only proceed with the replacement if "asmdeactivationoutcome" is "Yes" for all the disks:

cellcli -e list griddisk attributes name, status, asmdeactivationoutcome

c) It is also a good practice to get the ASM formatted output and review it for any offline disks:
How To Gather/Backup ASM Metadata In A Formatted Manner? (Doc ID 470211.1)

3. Check the value for ASM_POWER_LIMIT attribute. If it is 0, then set it to a non zero value so that after the disk replacement rebalance can happen and the newly added disks will have data written on them.
alter system set asm_power_limit=4;

4. Make sure there are no rebalance operation running on any instance on any node:

select * from gv$asm_operation;

Above query should return 0 rows. Only when it returns 0 rows can we conclude that there is no rebalance running. If there are any rows, we must wait until the rebalance completes and the query returns no rows.

Please note that we are using gv$ view which will give data from all the ASM instances in the rack. Querying v$ is not sufficient, as the rebalance may be running on a different ASM instance.

5. Identify the ASM griddisks which are residing on the physical disk to be replaced:

a. On the cell node execute following:

- Find the lun number "0_11" in the list physicaldisk detail output
# cellcli -e list physicaldisk detail
.....
luns: 0_11
.....

- Find the "name" associated to this LUN using the following syntax
# cellcli -e list lun attributes name where id=0_11

- Find the celldisk created for this lun (physicaldisk)
# cellcli -e list celldisk where lun=0_11
CD_11_dmorlcel02 normal

- Find the griddisks created for this celldisk
# cellcli -e list griddisk where celldisk=CD_11_dmorlcel02
DATA_CD_11_dmorlcel02 active
RECO_CD_11_dmorlcel02 active
DBFS_DG_CD_11_dmorlcel02

b. Execute following on any db node connecting to the ASM instance:

SQL> select name, substr(path,1,50) , mount_status, header_status, state from v$asm_disk where name in ('DATA_CD_11_DMORLCEL02','RECO_CD_11_DMORLCEL02');

NAME SUBSTR (PATH,1,50) MOUNT_S HEADER_STA STATE
------------------------------ ---------------------------------------- ------
DATA_CD_11_DMORLCEL02 o/192.168.10.6/DATA_CD_11_dmorlcel02 CACHED MEMBER NORMAL
RECO_CD_11_DMORLCEL02 o/192.168.10.6/RECO_CD_11_dmorlcel02 CACHED MEMBER NORMAL
DBFS_DG_CD_11_DMORLCEL02 o/192.168.10.6/DBFS_DG_CD_11_dmorlcel02 CACHED MEMBER NORMAL

6. Now drop the disk from the diskgroup(s):

alter diskgroup DATA drop disk 'DATA_CD_11_DMORLCEL02';
alter diskgroup RECO drop disk 'RECO_CD_11_DMORLCEL02';
alter diskgroup DBFS_DG drop disk 'DBFS_DG_CD_11_DMORLCEL02';

7. Wait until the Oracle ASM disks associated with the grid disks on the bad disk have been successfully dropped and the rebalance has completed by querying the V$ASM_DISK_STAT and the GV$ASM_OPERATION views. The rebalance is completed once GV$ASM_OPERATION returns no rows.

V$ASM_DISK_STAT displays disk statistics in the same way that V$ASM_DISK does, but without performing discovery of new disks. This results in a less expensive operation. However, since discovery is not performed, the output of this view does not include any data about disks that are new to the system.

8. Mark this disk at the hardware level to be serviced by turning on the amber locator LED:

cellcli -e 'alter physicaldisk disk_name serviceled on'

Here disk_name should be replaced by the name of the physical disk '20:11'.

This step will help in identifying the disk to be replaced. Please note that this command may not work in versions lower than 11.2.2.2, but it is not mandatory.

9. Now replace the disk at the hardware level. The service LED will be illuminated for this disk as a result of the previous step.

10. After replacing the disk, the Exadata software should automatically create the LUN, celldisk and griddisks. You can tail the cell alert.log and ASM alert.log to see this activity. However, the disks will not be added back to the ASM diskgroups as we did not use FORCE option while dropping them. So we need to add them manually. Run the following from ASM:

alter diskgroup DATA add disk 'o/192.168.10.6/DATA_CD_11_dmorlcel02';

alter diskgroup RECO add disk 'o/192.168.10.6/RECO_CD_11_dmorlcel02';

alter diskgroup DBFS_DG add disk 'o/192.168.10.6/DBFS_DG_CD_11_dmorlcel02';

-- Location of the cell alert.log file:
$CELLTRACE/alert.log.

-- Location of the ASM alert.log
<diagnostic_dest>/diag/asm/+asm/+ASM2/trace

11. Check until the rebalance completes using followingquery:

select * from gv$asm_operation;

12. Once the rebalance is over, check the status of the ASM disks again and verify that everything is fine. We expect all disks to have a mount_status of ONLINE and mode_status of CACHED:

SQL> select name, substr(path,1,50) , mount_status, header_status, state from v$asm_disk where name in ('DATA_CD_11_DMORLCEL02','RECO_CD_11_DMORLCEL02');

13. When a new disk is inserted, the "Disk Cache Policy" may be set to "Disk's Default".

Ideally it should be 'Disabled' otherwise the exachk utility will report a failure. Run the following change this value after a new disk is inserted.

# /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -DisDskCache -LAll -a0

Refer to the following note for more information:
Exachk Fail With Disk Cache Policy is Not Set to Disabled On One or More Storage Server (Doc ID 1347716.1)

References

<NOTE:470211.1> - How To Gather & Backup ASM/ACFS Metadata In A Formatted Manner version 10.1, 10.2, 11.1, 11.2 and 12.1?
<NOTE:1347716.1> - Exachk Fail With Disk Cache Policy is Not Set to Disabled On One or More Storage Server

Attachments

This solution has no attachment