How to Expand the Zero Data Loss Recovery Appliance Storage Cells

Asset ID:	1-71-2105482.1
Update Date:	2018-05-25
Keywords:

Solution Type Technical Instruction Sure

Solution 2105482.1 : How to Expand the Zero Data Loss Recovery Appliance Storage Cells

Applies to:

Zero Data Loss Recovery Appliance X4 Hardware - Version All Versions and later
Zero Data Loss Recovery Appliance X5 Hardware - Version All Versions and later
Zero Data Loss Recovery Appliance Software - Version 12.1.0.1.0 to 12.1.1.1.1
Linux x86-64

Goal

Add new storage cells to an existing Zero Data Loss Recovery Appliance and grow the Storage Location.

Solution

These steps assume you have already re imaged and configured the IP addresses on the new cell server(s).

IMPORTANT!!!

Expanding Storage Cells for ZDLRA 12.2.1.1.1

For ZDLRA 12.2.1.1.1, RACLI is used for ALL storage cell expansion work.

Please follow the process indicated in the ZDLRA Owner's Guide:

https://docs.oracle.com/cd/E88198_01/AMOGD/app_racli.htm#AMOGD-GUID-6D9FBAF1-19B4-40C4-BD78-20A0891DE6AA

Expanding Storage Cells for ZDLRA 12.1.1.1.X

Logic For Diskgroup (Add Vs Create)

IMPORTANT!!!

The Recovery Appliance has a single DELTA diskgroup by default. If you expand a Recovery Appliance with 3 or more cells of a larger disk size then your current appliance has then you will have DELTA2 as well.

For example:
If you have a base rack ZDLRA with 4 TB disks initially and you purchase 3 new cells which have 8 TB disks you have the original DELTA with 4 TB disks, and a new DELTA2 with all 3 cells of 8TB disks.

In this scenario, the 8TB disks can be utilized fully.

If you have a base rack ZDLRA with 4 TB disks and you purchase less then 3 new cells which include 8 TB disks you will add those new cells to the existing DELTA disk group with the same grid disk size as your existing cells.

In this scenario, only 4TB of the 8TB disks can be utilized.

Note:
Once added, the disks CANNOT be removed unless the ZDLRA is reimaged.

Logic for Diskgroup

Validate New Cells Version

Confirm that each new cell is the SAME OS version as the existing cells. This step should be run on ALL NEW CELLS.

#imageinfo -ver

If the NEW CELLS are different then the existing cell image version, you MUST re image the NEW CELLS to match

Configure Cell Details

The following section shows you how to configure the new cell details.

It requires access to an existing cell, or existing OEDA install reports and the new cells to be configured.

Gather Existing Cell Information

Login to an existing cell as root and run cellcli -e 'list cell detail'
Example:
[root@cellname01 ~]# cellcli -e 'list cell detail'
name: cellname01
bbuStatus: normal
cellVersion: OSS_12.1.2.1.3_LINUX.X64_151021
cpuCount: 24/24
diagHistoryDays: 7
fanCount: 8/8
fanStatus: normal
flashCacheMode: WriteBack
id: 1351NM505A
interconnectCount: 2
interconnect1: ib0
interconnect2: ib1
iormBoost: 0.0
ipaddress1: 192.168.40.217/21
ipaddress2: 192.168.40.218/21
kernelVersion: 2.6.39-400.264.1.el6uek.x86_64
locatorLEDStatus: off
makeModel: Oracle Corporation SUN SERVER X4-2L High Capacity
memoryGB: 95
metricHistoryDays: 7
notificationMethod: mail,snmp
notificationPolicy: critical,warning,clear
offloadGroupEvents:
offloadEfficiency: 500.5
powerCount: 2/2
powerStatus: normal
releaseImageStatus: success
releaseVersion: 12.1.2.1.3.151021
releaseTrackingBug: 21446295
smtpFrom: ZDLRA_name
smtpFromAddr: hostname01@domain.com
smtpPort: 25
smtpServer: internal-mail-router.domain.com
smtpToAddr: admin.user1@domain.com,admin.user2@domain.com
smtpUseSSL: FALSE
snmpSubscriber: host=hostname01.domain.com,port=3872,community=public
host=hostname02.domain.com,port=3872,community=public
host=ilm-asr1.domain.com,port=162,community=public,type=ASR
status: online
temperatureReading: 27.0
temperatureStatus: normal
upTime: 3 days, 13:19
usbStatus: normal
cellsrvStatus: running
msStatus: running
rsStatus: running
[root@cellname01 ~]#

Set the new cell attributes

This section you will set the new cell attributes on ALL NEW CELLS.

- alter cell smtpServer='internal-mail-router.domain.com';
- alter cell smtpFromAddr='hostname01@domain.com';
- alter cell smtpToAddr='admin.user1@domain.com,admin.user2@domain.com ';
- alter cell notificationPolicy='critical,warning,clear';
- alter cell notificationMethod='mail,snmp';
- alter cell snmpSubscriber=((host='ilm-asr1.domain.com',port=162,community=public,type=ASR))
- alter cell smtpFrom='ZDLRA_name;
- alter cell smtpUseSSL='FALSE';
- alter cell smtpPort=25;

Configure Cell Disks and Grid Disks

Create the celldisks

This is run on ALL NEW CELLS

CellCLI> CREATE CELLDISK ALL

From an existing cell check the size of the griddisks (DELTA & CATALOG)

This is run on an EXISTING cell to get information for future steps.

CellCLI> list griddisk detail

EXAMPLE:

CellCLI> list griddisk detail

name: CATALOG_CD_02_cellname01

asmDiskGroupName: CATALOG

asmDiskName: CATALOG_CD_02_CELLNAME01

asmFailGroupName: CELLNAME01

availableTo:

cachedBy: "FD_14_cellname01, FD_15_cellname01, FD_13_cellname01, FD_12_cellname01"

cachingPolicy: default

cellDisk: CD_02_cellname01

comment: "Cluster cluzdlra1 diskgroup CATALOG"

creationTime: 2015-07-30T01:16:47-07:00

diskType: HardDisk

errorCount: 0

id: 7d71bda3-f8b8-44ed-abe7-098755b8ff02

offset: 3.481719970703125T

size: 160G

status: active

(TRUNCATED!!!)

***For DELTA sizing look for griddisk sizing for cells with name starting with DELTA.***

name: DELTA_CD_00_cellname01

asmDiskGroupName: DELTA

asmDiskName: DELTA_CD_00_CELLNAME01

asmFailGroupName: CELLNAME01

availableTo:

cachedBy: "FD_00_cellname01, FD_01_cellname01, FD_02_cellname01, FD_03_cellname01"

cachingPolicy: default

cellDisk: CD_00_cellname01

comment: "Cluster cluzdlra1 diskgroup DELTA"

creationTime: 2015-07-30T01:16:49-07:00

diskType: HardDisk

errorCount: 0

id: 00d0931c-ff29-4e03-933e-ca5e8c4996d5

offset: 32M

size: 6.763671875T

status: active

Create the Griddisks with the sizes from the existing cell

Using the values from your list griddisk detail, create the new grid disks on ALL NEW CELLS

Note:

Create new griddisk sizes to match the griddisk sizes of the existing cells if the cells are the same size.

Default Griddisk sizes:

4TB Cells griddisk size = 3.4248046875T

8TB Cells griddisk size = 6.763671875T

CellCLI> create griddisk all HARDDISK prefix=DELTA, size=6.763671875T

CellCLI> create griddisk all HARDDISK prefix=CATALOG, size=160G

Enable WriteBack Flash Cache Mode

Drop the flashcache

CellCLI> drop flashcache
Flash cache scah43celadm16_FLASHCACHE successfully dropped

CellCLI>

Check the status of ASM if the grid disks go OFFLINE

(Command should return 'Yes' for the grid disks being listed:)

CellCLI> list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

Deactivate the griddisk on the cell

CellCLI> alter griddisk all inactive

Shutdown cellsrv service

CellCLI> alter cell shutdown services cellsrv

Change the flashcachemode Parameter to writeback

CellCLI> alter cell flashCacheMode=WriteBack

Restart the cellsrv service

CellCLI> alter cell startup services cellsrv

Reactivate the griddisks on the cell

CellCLI> alter griddisk all active

Recreate the flashcache

CellCLI> create flashcache all

Ensure the cell is 'Writeback' mode

CellCLI> list cell attributes flashCacheMode

Drop the first two Griddisks of CATALOG

(Deactivate the griddisks on the cell)

CellCLI> alter griddisk all inactive

Drop the Griddisks

Run this on each NEW CELL. Replace <cellname> with the NEW CELLS name.

CellCLI> drop griddisk CATALOG_CD_00_<cellname>
CellCLI> drop griddisk CATALOG_CD_01_<cellname>

Reactivate the Griddisks

CellCLI> alter griddisk all active

Update the DB Nodes for ALL NEW CELLS

Edit cellip.ora file (as oracle user)

On each existing compute node, Add ALL NEW CELLS IB IP information.

Again you must edit this file on ALL existing compute node.

- You will be addiing the NEW CELLS IB IP informatoin.

$ cd /etc/oracle/cell/network-config

$ vi cellip.ora

Add IB ip addresses for ALL the NEW cells to be added are added to this file

EXAMPLE:

[oracle@hostname01 network-config]$ cat cellip.ora

cell="192.168.40.223;192.168.40.224"

cell="192.168.40.225;192.168.40.226"

cell="192.168.40.227;192.168.40.228"

cell="192.168.40.229;192.168.40.230"

Edit /etc/hosts to include the new host (as root user)

NOTE:

DO NOT COPY THE '/etc/hosts' FILE TO OTHER SERVERS. EDIT EACH HOST FILE!!!

You will edit the /etc/hosts files on ALL hosts (compute and storage nodes). Adding the new IB addresses and host information.

EXAMPLE

[oracle@hostname01 ~]$ cat /etc/hosts

#### BEGIN Generated by Exadata. DO NOT MODIFY ####

127.0.0.1 localhost.localdomain localhost

# 192.168.40.133 hostname01-priv1.domain.com hostname01-priv1

# 192.168.40.134 hostname01-priv2.domain.com hostname01-priv2

10.133.62.67 host1ingest01.domain.com host1ingest01

10.133.241.200 host1repl01@domain.com host1repl01

10.133.40.127 hostname01.domain.com hostname01

#### END Generated by Exadata ####

#### BEGIN Added by Configuration Utility ####

192.168.40.133 hostname01-priv1 hostname01-priv1.domain.com

192.168.40.134 hostname01-priv2 hostname01-priv2.domain.com

192.168.40.135 hostname02-priv1 hostname02-priv1.domain.com

192.168.40.136 hostname02-priv2 hostname02-priv2.domain.com

192.168.40.217 cellname01-priv1 cellname01-priv1.domain.com

192.168.40.218 cellname01-priv2 cellname01-priv2.domain.com

192.168.40.219 cellname02-priv1 cellname02-priv1.domain.com

192.168.40.220 cellname02-priv2 cellname02-priv2.domain.com

192.168.40.221 cellname03-priv1 cellname03-priv1.domain.com

192.168.40.222 cellname03-priv2 cellname03-priv2.domain.com

192.168.40.223 cellname04-priv1 cellname04-priv1.domain.com

192.168.40.224 cellname04-priv2 cellname04-priv2.domain.com

192.168.40.225 cellname05-priv1 cellname05-priv1.domain.com

192.168.40.226 cellname06-priv2 cellname06-priv2.domain.com

192.168.40.227 cellname07-priv1 cellname07-priv1.domain.com

192.168.40.228 cellname07-priv2 cellname07-priv2.domain.com

#### END Added by Configuration Utility ####

[oracle@hostname01 ~]$

Verify accessibility of all grid disks (run as the root user or the oracle user.)

$ /u01/app/12.1.0.2/grid/bin/kfod disks=all dscvgroup=true

Diskgroup Logic Summary

Am I creating a new disk group or adding?

- Are my current cells the same disk size as my new cells?

Yes:

- Updating (Expanding) existing disk groups

No:

- Do I have 3 or more new cells to add?

Yes:

- - Adding a new Disk Group +DELTAn

No:

- - Updating (Expanding) existing disk groups

Set the Environment Variables (oracle user)

$ .oraenv

ORACLE_SID = [oracle] ? +ASM1

The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle

Add the Griddisks (DELTA & CATALOG) to the Diskgroup (as sysasm)

This will be run to add the new CELLS Diskgroups

$ sqlplus / as sysasm

EXAMPLE:
(Add all cells separated by comma then run rebalance as shown below)

For each new cell that you have you will have a new line to each alter statement.

Note: To create a new ASM DELTAn Diskgroup, use this command structure (below):

create diskgroup DELTA2
NORMAL REDUNDANCY
DISK 'o/*/DELTA2*'
ATTRIBUTE 'cell.smart_scan_capable'='TRUE',
'au_size'='4194304',
'compatible.rdbms'='12.1.0.2.0',
'compatible.asm'='12.1.0.2.0';

Note: To add the griddisks to the existing ASM DELTA Diskgroup use this command structure (below):

SQL> ALTER DISKGROUP DELTA ADD DISK

2> 'o/*/DELTA*NEW_CELLNAME01'

3> rebalance power 4;

Add the CATALOG Griddisks from ALL NEW cells to ASM Diskgroup

NOTE: Multiple cells are separated with a comma(,):

Example:

SQL> alter diskgroup CATALOG add disk

2 'o/*/CATALOG*NEW_CELL1*',

3 'o/*/CATALOG*NEW_CELL2*'

4 rebalance power 4;

SQL> alter diskgroup CATALOG add disk

2 'o/*/CATALOG*NEW_CELL1*'

3 rebalance power 4;

Confirm that the Rebalance is Running

SQL> SELECT * FROM GV$ASM_OPERATION WHERE STATE = 'RUN';

INST_ID GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE CON_ID

---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- -------------------------------------------- ----------
1 2 REBAL REBALANCE RUN 11 11 26235 23028878 10283 2236 0
2 1 REBAL REBALANCE RUN 11 11 19215 352254 12985 25 0

Confirm Rebalance is complete

SQL> SELECT * FROM GV$ASM_OPERATION WHERE STATE = 'RUN';

no rows selected

SQL>

Confirm that the groups are online

SQL> select group_number,failgroup,mode_status,count(*) from v$asm_disk group by group_number,failgroup,mode_status;

GROUP_NUMBER FAILGROUP MODE_ST COUNT(*)
------------ ------------------------------ ------- ----------
1 CELLNAME01 ONLINE 10
1 CELLNAME02 ONLINE 10
1 CELLNAME03 ONLINE 10
1 CELLNAME04 ONLINE 10
2 CELLNAME01 ONLINE 12
2 CELLNAME02 ONLINE 12
2 CELLNAME03 ONLINE 12
2 CELLNAME04 ONLINE 12
8 rows selected.
SQL>

Finalize the ZDLRA Configurations

[root@hostname01 ~]# su - oracle

[oracle@hostname01 ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Sat Nov 28 19:50:30 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters and Automatic Storage Management options

SQL>

Perform ZDLRA specific customizations (Adding Storage to DELTA - ONLY!!!)

Update the db_recovery_file_dest to the new Storage Location

Use API call to Change the value of db_recovery_file_dest_size parameter

SQL> exec DBMS_RA_ADM.UPDATE_FRA

System altered.

SQL>

Increase the size of the RA_POOL tablespace.

SQL> exec DBMS_RA_ADM.CREATE_RA_POOL('+DELTA');

Update the SL_sizing Table Using the API Call

SQL> exec DBMS_RA_ADM.UPDATE_SL_TABLE;

PL/SQL procedure successfully completed.

Get the size of the Storage Location as RASYS USER

[oracle@hostname01 ~]$ sqlplus rasys/ra
SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 14 08:36:58 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Last Successful login time: Tue Jul 14 2015 08:00:16 -07:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit
Production
With the Partitioning, Real Application Clusters and Automatic Storage
Management options

SQL> column sl_space format 9999999999999999999 ;

Validate the size of Diskgroup (+DELTA) to be added to the Storage Location

SQL> select * from sl_sizing;

DISK_GROUP
--------------------------------------------------------------------------------
I SL_SPACE
- --------------------DELTA
Y 70771314735
SQL>

Update the Storage Locations to include the new storage

SQL> exec dbms_ra.update_storage_location (STORAGE_LOCATION_NAME => 'DELTA', STORAGE_LOCATION_DESTS => '+DELTA( 70771314735)');

PL/SQL procedure successfully completed.

Confirm that the storage location has increased in size

SQL> select total_space from ra_storage_location;

TOTAL_SPACE
-----------
67492.7852
Login as sysdba

Update the throttle to update the config values

Login ( / as sysdba)

SQL> exec dbms_ra_adm.update_throttle;

**********END (SINGLE DELTA CELL EXPANSION)************

Perform ZDLRA specific customizations (Creating a New DELTAn - ONLY!!!)

Mount the new diskgroup on the compute node 2

Login ( / as sysasm)

[oracle@hostname02 ~]# . oraenv

ORACLE_SID = [zdlra1] ? +ASM2

[oracle@hostname02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.1.0.2.0 Production on Sat Nov 28 19:50:30 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters and Automatic Storage Management options

SQL>

SQL> ALTER DISKGROUP DELTA2 MOUNT;

Diskgroup altered

Update the db_recovery_file_dest to the new Storage Location

Login ( / as sysdba)

Use API call to Change the value of db_recovery_file_dest_size parameter

SQL> exec DBMS_RA_ADM.UPDATE_FRA

System altered.
SQL>

Check the NEW value of db_recovery_file_dest_size parameter

SQL> show parameter db_recovery_file

NAME TYPE VALUE
------------------------------------ -----------
------------------------------
db_recovery_file_dest string +DELTA2
db_recovery_file_dest_size big integer 3072G

SQL> show parameter log_archive_dest_3;

NAME TYPE VALUE
------------------------------------ -----------
------------------------------
log_archive_dest_3 string location=+DELTA2 valid_for=(ST
ANDBY_LOGFILE,ALL_ROLES)
log_archive_dest_30 string
log_archive_dest_31 string
SQL>
Increase the size of the RA_POOL tablespace.

Increase the size of the RA_POOL tablespace.

SQL> exec DBMS_RA_ADM.CREATE_RA_POOL('+DELTA2');

Update the SL_sizing Table Using the API Call

SQL> exec DBMS_RA_ADM.UPDATE_SL_TABLE;

PL/SQL procedure successfully completed.

Get the size of the Storage Location as RASYS USER

[oracle@scas10adm07 ~]$ sqlplus rasys/ra
SQL*Plus: Release 12.1.0.2.0 Production on Tue Jul 21 20:42:21 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Last Successful login time: Tue Jul 21 2015 20:03:25 -07:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters and Automatic Storage Management options

SQL> column sl_space format 9999999999999999999 ;

Validate the size of Diskgroup (DELTA2) to be added to the Storage Location

SQL> select * from sl_sizing;

DISK_GROUP
--------------------------------------------------------------------------------
I SL_SPACE
- --------------------
DELTA2
Y 105535871385600
DELTA
Y 52352324206592
SQL>

Update the Storage Locations to include the new storage

SQL> exec dbms_ra.update_storage_location (STORAGE_LOCATION_NAME => 'DELTA', STORAGE_LOCATION_DESTS => '+DELTA2( 105535871385600)');

PL/SQL procedure successfully completed.

Confirm the Expansion was Successful by checking the sizes for SL

SQL> select disk_group, sl_space from sl_sizing;

DISK_GROUP
--------------------------------------------------------------------------------
SL_SPACE
--------------------
DELTA2
105535871385600
DELTA
52352313720832
SQL>

SQL> select name, sl_key, total_space, used_space from rasys.ra_storage_location;

Update the throttle to update the config values

Login( / as sysdba)

SQL> exec dbms_ra_adm.update_throttle;

**********END (MULTI DELTAn CELL EXPANSION)************

Attachments

This solution has no attachment