How To Expand griddisks for 10TB(Exadata X7-2 Storage Cells)

Asset ID:	1-71-2375820.1
Update Date:	2018-03-28
Keywords:

Solution Type Technical Instruction Sure

Solution 2375820.1 : How To Expand griddisks for 10TB(Exadata X7-2 Storage Cells)

Applies to:

Zero Data Loss Recovery Appliance X7 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

The goal of this document is to provide instructions on how to correctly expand undersized X7(10TB) Storage cells to their appropriate size to allow for optimal utilization of space on the drives.

*Solution
Provide instructions on how to accomplish the goal.

Applies to ZDLRA X7 Hardware

Solution

1. Check if your ZDLRA X7 is affected by this issue

sqlplus / as sysdba
SQL> select max(os_mb) from v$asm_disk;
MAX(OS_MB)
----------
7474176

Note:
If the MAX(OS_MB) = 7474176, follow the steps in this note.

If the MAX(OS_MB) = 8875008, your system is not affected. You do NOT need to follow this note.

If you find that MAX(OS_MB) is any other number than 7474176 or 8875008, Contact Oracle Support.

2. Create a file with the list of 10TB cells

Note:
The file /root/cell_group contains the list of ALL cells in the environment.

[zdlraadm01 ~]# cat /root/cell_group
zdlraceladm06
zdlraceladm07
zdlraceladm08
zdlraceladm09
[root@zdlraadm01 ~]#

3. Setup SSH keys for the cells

[root@zdlraadm01 os_patch]# ssh-keygen -t rsa
[root@zdlraadm01 ~]# dcli -g /root/cell_group -k -l root

4. Expand the Griddisks to the new 10TB size

Note:

This step needs to be performed for each cell in the environment.

5. Verify the new Griddisk sizes for the 10TB cells

Note:

The new Griddisk size should be 8.4638671875T

[root@zdlraadm01 ~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes name,size \
> where name like \'DELTA.*\' "

6. Resize the ASM Diskgroup to include the new Griddisk sizes (On First Compute Node - as oracle user)

[oracle@ zdlraadm01 ~]$ . oraenv
ORACLE_SID = [zdlrax1] ? +ASM1

7. Confirm that rebalance of the Diskgroup is complete

sqlplus / as sysasm

SQL> set lines 250 pages 1000
SQL> col error_code form a10
SQL> SELECT dg.name, o.* FROM gv$asm_operation o, v$asm_diskgroup dg
2 WHERE o.group_number = dg.group_number;

SQL> SELECT * FROM GV$ASM_OPERATION WHERE STATE = 'RUN';

Note:
Until output says "no rows selected", do NOT begin the next step.

8. Confirm that the MAX(os_mb) value is now 8875008.

sqlplus / as sysdba

SQL> select max(os_mb) from v$asm_disk;

9. Update the Storage Location

In this step, you will use either 9.1 or 9.2 depending on the version of ZDLRA Software you are running.

9.1 Update Storage Location (Use ONLY for ZDLRA 12.1 Systems)

sqlplus / as sysdba

BEGIN

DBMS_RA_ADM.UPDATE_FRA;
DBMS_RA_ADM.CREATE_RA_POOL('+DELTA');
DBMS_RA_ADM.UPDATE_SL_TABLE;

END;

SELECT TO_CHAR(

'+'
|| disk_group
|| '('
|| sl_space
|| ')'
) SLSIZE

FROM rasys.sl_sizing

WHERE disk_group = 'DELTA';

Query output

SLSIZE
--------------------
+DELTA(xxxxxxxxxxxx)

Use output from select statement in the following API

sqlplus / as sysdba

BEGIN

RASYS.DBMS_RA.UPDATE_STORAGE_LOCATION('DELTA',
STORAGE_LOCATION_DESTS=>('+DELTA(xxxxxxxxxxxx)'));

END;

9.2 Update the Storage Location (Use ONLY for ZDLRA 12.2 Systems)

[root@zdlraadm01 bin]# /opt/oracle/RecoveryAppliance/bin/racli update storage_location --cells=zdlraceladm06,zdlraceladm07,zdlraceladm08,zdlraceladm09

10. Verify the total space of the Storage Location

SQL> select name, total_space from ra_storage_location;

Appendix: Actual Run Output

1. Check if your ZDLRA X7 is affected by this issue

sqlplus / as sysdba

SQL> select max(os_mb) from v$asm_disk;
MAX(OS_MB)
----------
7474176

Note:
If the MAX(OS_MB) = 7474176, follow the steps in this note.
If the MAX(OS_MB) = 8875008, your system is not affected.
If you find that MAX(OS_MB) is any other number than 7474176 or 8875008, Contact Oracle Support.

2. Create a file with the list of 10TB cells

Note:
The file /root/cell_group contains the list of ALL cells in the environment.

[zdlraadm01 ~]# cat /root/cell_group
zdlraceladm06
zdlraceladm07
zdlraceladm08
zdlraceladm09
[root@zdlraadm01 ~]#

3. Setup SSH keys for the cells

[root@zdlraadm01 os_patch]# ssh-keygen -t rsa
[root@zdlraadm01 ~]# dcli -g /root/cell_group -k -l root

4. Expand the Griddisks to the new 10TB size

Note:

This step needs to be performed for each cell in the environment.

[root@zdlraadm01 ~]# dcli -c zdlraceladm06 -l root "cellcli -e alter griddisk DELTA_CD_00_zdlraceladm06 \
> ,DELTA_CD_01_zdlraceladm06 \
> ,DELTA_CD_02_zdlraceladm06 \
> ,DELTA_CD_03_zdlraceladm06 \
> ,DELTA_CD_04_zdlraceladm06 \
> ,DELTA_CD_05_zdlraceladm06 \
> ,DELTA_CD_06_zdlraceladm06 \
> ,DELTA_CD_07_zdlraceladm06 \
> ,DELTA_CD_08_zdlraceladm06 \
> ,DELTA_CD_09_zdlraceladm06 \
> ,DELTA_CD_10_zdlraceladm06 \
> ,DELTA_CD_11_zdlraceladm06 \
> size=8667G "
zdlraceladm06: GridDisk DELTA_CD_00_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_01_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_02_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_03_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_04_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_05_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_06_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_07_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_08_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_09_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_10_zdlraceladm06 successfully altered
zdlraceladm06: GridDisk DELTA_CD_11_zdlraceladm06 successfully altered
[root@zdlraadm01 ~]#

[root@zdlraadm01 ~]# dcli -c zdlraceladm07 -l root "cellcli -e alter griddisk DELTA_CD_00_zdlraceladm07 \
> ,DELTA_CD_01_zdlraceladm07 \
> ,DELTA_CD_02_zdlraceladm07 \
> ,DELTA_CD_03_zdlraceladm07 \
> ,DELTA_CD_04_zdlraceladm07 \
> ,DELTA_CD_05_zdlraceladm07 \
> ,DELTA_CD_06_zdlraceladm07 \
> ,DELTA_CD_07_zdlraceladm07 \
> ,DELTA_CD_08_zdlraceladm07 \
> ,DELTA_CD_09_zdlraceladm07 \
> ,DELTA_CD_10_zdlraceladm07 \
> ,DELTA_CD_11_zdlraceladm07 \
> size=8667G "
zdlraceladm07: GridDisk DELTA_CD_00_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_01_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_02_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_03_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_04_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_05_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_06_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_07_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_08_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_09_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_10_zdlraceladm07 successfully altered
zdlraceladm07: GridDisk DELTA_CD_11_zdlraceladm07 successfully altered
[root@zdlraadm01 ~]#

[root@zdlraadm01 ~]# dcli -c zdlraceladm08 -l root "cellcli -e alter griddisk DELTA_CD_00_zdlraceladm08 \
> ,DELTA_CD_01_zdlraceladm08 \
> ,DELTA_CD_02_zdlraceladm08 \
> ,DELTA_CD_03_zdlraceladm08 \
> ,DELTA_CD_04_zdlraceladm08 \
> ,DELTA_CD_05_zdlraceladm08 \
> ,DELTA_CD_06_zdlraceladm08 \
> ,DELTA_CD_07_zdlraceladm08 \
> ,DELTA_CD_08_zdlraceladm08 \
> ,DELTA_CD_09_zdlraceladm08 \
> ,DELTA_CD_10_zdlraceladm08 \
> ,DELTA_CD_11_zdlraceladm08 \
> size=8667G "
zdlraceladm08: GridDisk DELTA_CD_00_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_01_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_02_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_03_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_04_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_05_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_06_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_07_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_08_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_09_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_10_zdlraceladm08 successfully altered
zdlraceladm08: GridDisk DELTA_CD_11_zdlraceladm08 successfully altered
[root@zdlraadm01 ~]#

[root@zdlraadm01 ~]# dcli -c zdlraceladm09 -l root "cellcli -e alter griddisk DELTA_CD_00_zdlraceladm09 \
> ,DELTA_CD_01_zdlraceladm09 \
> ,DELTA_CD_02_zdlraceladm09 \
> ,DELTA_CD_03_zdlraceladm09 \
> ,DELTA_CD_04_zdlraceladm09 \
> ,DELTA_CD_05_zdlraceladm09 \
> ,DELTA_CD_06_zdlraceladm09 \
> ,DELTA_CD_07_zdlraceladm09 \
> ,DELTA_CD_08_zdlraceladm09 \
> ,DELTA_CD_09_zdlraceladm09 \
> ,DELTA_CD_10_zdlraceladm09 \
> ,DELTA_CD_11_zdlraceladm09 \
> size=8667G "
zdlraceladm09: GridDisk DELTA_CD_00_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_01_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_02_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_03_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_04_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_05_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_06_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_07_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_08_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_09_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_10_zdlraceladm09 successfully altered
zdlraceladm09: GridDisk DELTA_CD_11_zdlraceladm09 successfully altered
[root@zdlraadm01 ~]#

5. Verify the new Griddisk sizes for the 10TB cells

Note:

The new Griddisk size should be 8.4638671875T

[root@zdlraadm01 ~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes name,size \
> where name like \'DELTA.*\' "
zdlraceladm06: DELTA_CD_00_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_01_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_02_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_03_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_04_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_05_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_06_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_07_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_08_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_09_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_10_zdlraceladm06 8.4638671875T
zdlraceladm06: DELTA_CD_11_zdlraceladm06 8.4638671875T
zdlraceladm07: DELTA_CD_00_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_01_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_02_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_03_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_04_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_05_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_06_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_07_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_08_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_09_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_10_zdlraceladm07 8.4638671875T
zdlraceladm07: DELTA_CD_11_zdlraceladm07 8.4638671875T
zdlraceladm08: DELTA_CD_00_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_01_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_02_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_03_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_04_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_05_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_06_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_07_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_08_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_09_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_10_zdlraceladm08 8.4638671875T
zdlraceladm08: DELTA_CD_11_zdlraceladm08 8.4638671875T
zdlraceladm09: DELTA_CD_00_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_01_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_02_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_03_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_04_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_05_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_06_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_07_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_08_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_09_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_10_zdlraceladm09 8.4638671875T
zdlraceladm09: DELTA_CD_11_zdlraceladm09 8.4638671875T
[root@zdlraadm01 ~]#

6. Resize the ASM Diskgroup to include the new Griddisk sizes (On First Compute Node - as oracle)
salplus / as sysasm

[root@zdlraadm01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@zdlraadm01 ~]# su - oracle
[oracle@zdlraadm01 ~]$ . oraenv
ORACLE_SID = [zdlrax7b1] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[oracle@zdlraadm01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Thu Mar 1 13:26:21 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL>

SQL> ALTER DISKGROUP DELTA RESIZE ALL;
Diskgroup altered.
SQL>

7. Confirm that rebalance of the Diskgroup is complete

sqlplus / as sysasm

SQL> set lines 250 pages 1000
SQL> col error_code form a10
SQL> SELECT dg.name, o.* FROM gv$asm_operation o, v$asm_diskgroup dg
2 WHERE o.group_number = dg.group_number;
no rows selected
SQL>
SQL> SELECT * FROM GV$ASM_OPERATION WHERE STATE = 'RUN';
no rows selected
SQL>

Note:
Until output shows "no rows selected", do NOT begin the next step.

8. Confirm that the MAX(os_mb) value is now 8875008

sqlplus / as sysdba

SQL> select max(os_mb) from v$asm_disk;
MAX(OS_MB)
----------
8875008
SQL>

9. Update the Storage Location

In this step, you will use either 9.1 or 9.2 depending on the version of ZDLRA Software you are running.

9.1 Update Storage Location (Use ONLY for ZDLRA 12.1 Systems)

sqlplus / as sysdba

SQL> BEGIN
2 DBMS_RA_ADM.UPDATE_FRA;
3 DBMS_RA_ADM.CREATE_RA_POOL('+DELTA');
4 DBMS_RA_ADM.UPDATE_SL_TABLE;
5 END;
6 /

PL/SQL procedure successfully completed.

SQL>

SQL> SELECT TO_CHAR(
'+'
|| disk_group
|| '('
|| sl_space
|| ')'
) SLSIZE
FROM rasys.sl_sizing
WHERE disk_group = 'DELTA'; 2 3 4 5 6 7 8 9

SLSIZE
--------------------------------------------------------------------------------
+DELTA(132680234565632)

SQL>

Use output from select statement in the following API

SQL> BEGIN
2 RASYS.DBMS_RA.UPDATE_STORAGE_LOCATION('DELTA',STORAGE_LOCATION_DESTS=>('+DELTA(132680234565632)'));
3 END;
4 /

PL/SQL procedure successfully completed.

SQL>

SQL> select name, total_space from ra_storage_location;

NAME
--------------------------------------------------------------------------------
TOTAL_SPACE
-----------
DELTA
123567.621

SQL>

9.2 Update Storage Location (Use ONLY for ZDLRA 12.2 Systems)

[root@zdlraadm01 bin]# ./racli update storage_location --cells=zdlraceladm06,zdlraceladm07,zdlraceladm08,zdlraceladm09
Created log file /opt/oracle.RecoveryAppliance/log/ra_addstoragelocation_20180301.1342.log

Getting physical disk info...
Disk sizes validation: Matched!
Checking the rebalance
Updating Storage Location...
Update Storage Location: - Complete
Extending the Storage Location...
Extend Storage Location: - Complete
Updating the throttle...
Update throttle: - Complete
AddStorageLocation - Complete

ZDLRA Cell Expansion - Complete!
[root@zdlraadm01 bin]#

10. Verify the total space of the Storage Location

sqlplus / as sysdba

SQL> select name, total_space from ra_storage_location;
NAME
--------------------------------------------------------------------------------
TOTAL_SPACE
-----------
DELTA
164614.418
SQL>

Attachments

This solution has no attachment