Exadata X3-2 Hardware
 
Oracle Exadata Hardware
 
Exadata X4-2 Hardware
 
Exadata Database Machine X2-8
 
Exadata X3-8 Hardware
 
Exadata Database Machine X2-2 Hardware
 
Exadata Database Machine V2
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

Applies to:

Goal

In Exadata Env, One voting disk suddenly went offline. This voting disk went offline due to underline grid disk went offline.

oracle@exa1101:[+ASM1]/u01/app/ora...asm/+ASM1/trace# crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 3342c3c5a8d84f1abfc09b3c9de3a85f (o/192.168.10.3/DBFS_DG_CD_02_exa11celm01) [DBFS_DG]
2. OFFLINE 9fc1636a00714f59bf1bf9638a57ebff (o/192.168.10.5/DBFS_DG_CD_02_exa11celm03) [DBFS_DG]
3. ONLINE 9292bf77871d4fcabf62198c4a3ac00c (o/192.168.10.4/DBFS_DG_CD_02_exa11celm02) [DBFS_DG]
Located 3 voting disk(s).

This is also reported in RAC Alertlog files.

[/u01/app/11.2.0.3/grid/bin/oraagent.bin(73440)]CRS-5818:Aborted command 'check' for resource 'ora.prbi10e11.prbi10_batch.svc'. Details at (:CRSAGF00113:) {0:5:2} in /u01/app/11.2.0.3/grid/log/exa1101/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2013-11-16 05:48:01.383
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(73440)]CRS-5818:Aborted command 'check' for resource 'ora.prwh20e11.prwh20_bck1.svc'. Details at (:CRSAGF00113:) {0:5:2} in /u01/app/11.2.0.3/grid/log/exa1101/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2013-11-16 05:48:01.383
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(73440)]CRS-5014:Agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0.3/grid/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/u01/app/11.2.0.3/grid/log/exa1101/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2013-11-16 11:40:39.132
[cssd(12332)]CRS-1604:CSSD voting file is offline: o/192.168.10.5/DBFS_DG_CD_02_exa11celm03; details at (:CSSNM00058:) in /u01/app/11.2.0.3/grid/log/exa1101/cssd/ocssd.log.
2013-11-18 09:37:10.912
[crsd(13181)]CRS-2773:Server 'exa1101' has been removed from pool 'ora.prbo10e11_PRBO10_BCK1'.
2013-11-18 09:37:10.913
[crsd(13181)]CRS-2773:Server 'exa1102' has been removed from pool 'ora.prbo10e11_PRBO10_BCK1'.

In this document, we explain what are the ways to make voting disk online in this kind of situation.

Solution

There are two ways to make voting disk online in Exadata. Though, suggested method also apply on non Exadata env as well.

1. Drop grid disk having offline voting disk and voting disk will move automatically to another grid disk.
2. Move voting disk from one disk group to another disk group.

1. Drop grid disk having offline voting disk and voting disk will move automatically to another grid disk:

Drop grid disk having offline voting disk and voting disk will automatically move to some other grid disk in same disk group. In Exadata each fail group has minimum 10 disks, So disk will move to any of the existing disks. Since, you have redundency at ASM level so there is no data loss. Below are the steps:-

root@dmorldb07:/u01/app/11.2.0.3/grid/bin# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 0876580404f14f00bf5e5b5a4f06c230 (o/192.168.10.19/DBFS_DG_CD_02_dmorlcel11) [DBFS_DG]
2. OFFLINE 488de3c2214f4f18bff5242d725c982a (o/192.168.10.20/DBFS_DG_CD_04_dmorlcel12) [DBFS_DG]
3. ONLINE 5798c54ac29f4f5cbfc695e6ea33ef48 (o/192.168.10.21/DBFS_DG_CD_05_dmorlcel13) [DBFS_DG]
Located 3 voting disk(s).

SQL> alter diskgroup DBFS_DG drop disk DBFS_DG_CD_04_dmorlcel12 REBALANCE POWER 11;

Diskgroup altered.

root@dmorldb07:/u01/app/11.2.0.3/grid/bin# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 0876580404f14f00bf5e5b5a4f06c230 (o/192.168.10.19/DBFS_DG_CD_02_dmorlcel11) [DBFS_DG]
2. ONLINE 5798c54ac29f4f5cbfc695e6ea33ef48 (o/192.168.10.21/DBFS_DG_CD_05_dmorlcel13) [DBFS_DG]
3. ONLINE a86876efe1aa4f77bfc7e4236539da41 (o/192.168.10.20/DBFS_DG_CD_08_dmorlcel12) [DBFS_DG]

Voting disk has moved from o/192.168.10.20/DBFS_DG_CD_04_dmorlcel12 to o/192.168.10.20/DBFS_DG_CD_08_dmorlcel12 and also online.

2. Move voting disk from one disk group to another disk group:

We can move all voting disks from one disk group to another disk group. Before that, we need to know one basic behaviour of voting disk. if disk group having voting disk has normal redundency, then number of voting disk will be three. If disk group having voting disk has higher redundency, then number of voting disk will be five and each voting disk reside on one fail group.

In Exadata, Each cell server is a fail group. This means Exadata Eighth and Quarter Rack which has only three fail groups (cell server) can have high redundency disk group but can't afford voting disk on these disk groups. So, we need to make a different disk group with normal redundency.

I try this and got below error message:-

ERROR: Voting file allocation failed for group RECO_EXA11
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_77630.trc:
ORA-15274: Not enough failgroups (5) to create voting files
...
Wed Nov 20 16:51:31 2013
NOTE: Creating voting files in diskgroup DATA_EXA11
Wed Nov 20 16:51:31 2013
NOTE: Voting File refresh pending for group 1/0x8a08a35d (DATA_EXA11)
...
ERROR: Voting file allocation failed for group DATA_EXA11
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_82570.trc:
ORA-15274: Not enough failgroups (5) to create voting files

However, There is no risk involve in this, since voting files restore back on existing disk group, but before moving voting disk make sure you have enough fail groups. Below is the std rule in Exadata:-

1. Voting disk on normal redundency disk group should have minimum three cell servers.
2. Voting disk on high redundency disk group should have minimum five cell servers.

Steps to move voting disk from one disk group to another disk group:-

[grid@odain2 ~]$ crsctl query css votedisk
[grid@odain2 ~]$ crsctl replace votedisk +DATA

Now all voting disk has moved to +DATA disk group and are online.

References

This solution has no attachment