How to Replace an Oracle Database Appliance X6-2 Small/Medium/Large NVME drive

Asset ID:	1-71-2158779.1
Update Date:	2017-06-21
Keywords:

Solution Type Technical Instruction Sure

Solution 2158779.1 : How to Replace an Oracle Database Appliance X6-2 Small/Medium/Large NVME drive

Applies to:

Oracle Database Appliance X6-2 Hardware - Version All Versions and later
x86_64

Goal

How to Replace an NVME drive on an Oracle Database Appliance X6-2 Small/Medium/Large

Solution

DISPATCH INSTRUCTIONS:

NVME drives are CRUs (Customer Replaceable Units) and drives should be sent to the site for replacement by the customer.

TIME ESTIMATE: 30 minutes

TASK COMPLEXITY: 0

PROBLEM OVERVIEW:
Steps to replace a failed NVME drive in ODA X6-2S and ODA X6-2M and ODA X6-2L.

INSTRUCTIONS:

1) Identify the disk drive that needs replacement, below is an example from a FAILED drive. In this example, we are replacing disk pd_00.

# odaadmcli show disk
NAME PATH TYPE STATE STATE_DETAILS

pd_00 /dev/nvme0n1 NVD FAILED DiskRemoved
pd_01 /dev/nvme1n1 NVD ONLINE Good

2) Make sure that the Good disk is online in ASM:

# odaadmcli stordiag pd_01
Node Name : odax6-2m-sin06-a
Test : Diagnostic Test Description

1 : OAK Check
NAME PATH TYPE STATE STATE_DETAILS
pd_01 /dev/nvme1n1 NVD ONLINE Good

2 : ASM Check
ASM Disk Status : group_number state mode_s mount_s header_s
/dev/NVD_S01_S2LHNAAH503564p2 : 2 NORMAL ONLINE CACHED MEMBER <<<<<<<<<< should be ONLINE CACHED MEMBER
/dev/NVD_S01_S2LHNAAH503564p1 : 1 NORMAL ONLINE CACHED MEMBER <<<<<<<<<< should be ONLINE CACHED MEMBER

3 : Print NVMe critical warning and percentage-used
nvme-critical-warning = 0
nvme-percent-used = 0

4 : Check Partition using parted
Check using active device path: /dev/nvme1n1
Partition check on device /dev/nvme1n1 : PASS
Partition list found by parted for active device path: /dev/nvme1n1
1 2097kB 1280GB 1280GB primary
2 1280GB 3201GB 1920GB primary

5 : Fishwrap
NVMe controller: nac(/SYS/DBP/NVME1) PCI address(1b:00.0) slot_number(11) model_number (MS1PC2DD3ORA3.2T)

6 : NVMe information from nvmeadm tool
SUNW-NVME-2
PCI Vendor ID: 144d
Serial Number: S2LHNAAH503564
Model Number: MS1PC2DD3ORA3.2T
Firmware Revision: KPYA8R3Q

7 : Version Information
oda-hw-mgmt : 12.1.2.8.0
kernel : 2.6.39-400.276.1.el6uek.x86_64
NVMe Firmware : 8R3Q

8 : OAK Conf Parms
Device : rq_affinity max_sectors_kb nr_requests read_ahead_kb scheduler
/dev/nvme1n1 : 0 128 128 128 none

3) Make sure the NVME drive is powered off with the odaadmcli power command:

# odaadmcli power disk off pd_00
Disk 'pd_00' is already part of ASM

Are you sure you want to power OFF the disk? : 'pd_00'? [yes/no] :
yes
Powered OFF pd_00

# odaadmcli power disk off pd_00
Disk 'pd_00' already powered off

4) At this point, the blue led on the NVME disk should be on, and it is ok to now swap this disk with a new one.

NOTE* NVME disk0 (pd_00) is actually the drive in slot 2, and NVME disk1 (pd_01) is the drive in slot 3. Do not get this confused with the HDD drives in slot0 and slot1. See the illustration below:

driveslots

4A) Power on the disk with odaadmcli command

# odaadmcli power disk on pd_00
Powered ON pd_00

# odaadmcli show disk
NAME PATH TYPE STATE STATE_DETAILS

pd_00 /dev/nvme0n1 NVD ONLINE Good
pd_01 /dev/nvme1n1 NVD ONLINE Good

5) After the disk has been replaced, verify that it has been added to ASM.

If the disk has been automatically added, then the replacement procedure is complete. If not, move on to step 6.

# odaadmcli stordiag pd_00
Node Name : odax6-2m-sin06-a
Test : Diagnostic Test Description

1 : OAK Check
NAME PATH TYPE STATE STATE_DETAILS
pd_00 /dev/nvme0n1 NVD ONLINE Good

2 : ASM Check
ASM Disk Status : group_number state mode_s mount_s header_s
/dev/NVD_S00_S2LHNAAH503592p1 : 0 NORMAL ONLINE CLOSED MEMBER <<<<<<<<<<<<<<<<<<<<<<<<< if this does not say ONLINE CACHED MEMBER, move on to next step
/dev/NVD_S00_S2LHNAAH503592p2 : 0 NORMAL ONLINE CLOSED MEMBER <<<<<<<<<<<<<<<<<<<<<<<<< if this does not say ONLINE CACHED MEMBER, move on to next step

3 : Print NVMe critical warning and percentage-used
nvme-critical-warning = 0
nvme-percent-used = 0

4 : Check Partition using parted
Check using active device path: /dev/nvme0n1
Partition check on device /dev/nvme0n1 : PASS
Partition list found by parted for active device path: /dev/nvme0n1
1 2097kB 1280GB 1280GB primary
2 1280GB 3201GB 1920GB primary

5 : Fishwrap
NVMe controller: nac(/SYS/DBP/NVME0) PCI address(19:00.0) slot_number(10) model_number (MS1PC2DD3ORA3.2T)

6 : NVMe information from nvmeadm tool
SUNW-NVME-1
PCI Vendor ID: 144d
Serial Number: S2LHNAAH503592
Model Number: MS1PC2DD3ORA3.2T
Firmware Revision: KPYA8R3Q

7 : Version Information
oda-hw-mgmt : 12.1.2.8.0
kernel : 2.6.39-400.276.1.el6uek.x86_64
NVMe Firmware : 8R3Q

8 : OAK Conf Parms
Device : rq_affinity max_sectors_kb nr_requests read_ahead_kb scheduler
/dev/nvme0n1 : 1 128 128 128 none

6) Add the disk to ASM using the odaadmcli addasmdisk command. This step is needed only if step 5 output does have say ONLINE CACHED MEMBER (see output above).

# odaadmcli addasmdisk -all
Trying to sync disk status between ASM DiskGroup and OAKD
Display ASM diskgroup mount status
RECO MOUNTED
DATA MOUNTED
List of all disks not in sync betwen ASM and OAKD
/dev/NVD_S00_S2LHNAAH503592p1
/dev/NVD_S00_S2LHNAAH503592p2
Issuing SQL command to alter diskgroup, they may take some time...
[SUCCESS] : Out of sync disks belonging to DATA diskgroup are added back to DATA diskgroup
[SUCCESS] : Out of sync disks belonging to RECO diskgroup are added back to RECO diskgroup

6A) Confirm disk has been successfully added to ASM:

# odaadmcli stordiag pd_00
Node Name : odax6-2m-sin06-a
Test : Diagnostic Test Description

1 : OAK Check
NAME PATH TYPE STATE STATE_DETAILS
pd_00 /dev/nvme0n1 NVD ONLINE Good

2 : ASM Check
ASM Disk Status : group_number state mode_s mount_s header_s
/dev/NVD_S00_S2LHNAAH503592p1 : 1 NORMAL ONLINE CACHED MEMBER
/dev/NVD_S00_S2LHNAAH503592p2 : 2 NORMAL ONLINE CACHED MEMBER

3 : Print NVMe critical warning and percentage-used
nvme-critical-warning = 0
nvme-percent-used = 0

5 : Fishwrap
NVMe controller: nac(/SYS/DBP/NVME0) PCI address(19:00.0) slot_number(10) model_number (MS1PC2DD3ORA3.2T)

6 : NVMe information from nvmeadm tool
SUNW-NVME-1
PCI Vendor ID: 144d
Serial Number: S2LHNAAH503592
Model Number: MS1PC2DD3ORA3.2T
Firmware Revision: KPYA8R3Q

7 : Version Information
oda-hw-mgmt : 12.1.2.8.0
kernel : 2.6.39-400.276.1.el6uek.x86_64
NVMe Firmware : 8R3Q

8 : OAK Conf Parms
Device : rq_affinity max_sectors_kb nr_requests read_ahead_kb scheduler
/dev/nvme0n1 : 1 128 128 128 none

6b) Alternatively, you can enter ASM instance to confirm the disk is online:

SQL> select group_number,disk_number,mount_status,state,name from v$asm_disk;

GROUP_NUMBER DISK_NUMBER MOUNT_S STATE NAME
------------ ----------- ------- -------- ------------------------------
2 2 CACHED NORMAL SSD_QRMDSK_P2
1 2 CACHED NORMAL SSD_QRMDSK_P1
1 3 CACHED NORMAL NVD_S01_S2LHNAAH503567P1
2 3 CACHED NORMAL NVD_S01_S2LHNAAH503567P2
2 1 CACHED NORMAL NVD_S00_S2LHNAAH501760P2
1 1 CACHED NORMAL NVD_S00_S2LHNAAH501760P1

REFERENCE INFORMATION:

Oracle Dabase Server X6-2S/M/L Documentation

http://docs.oracle.com/cd/E83239_01/index.htm

https://docs.oracle.com/cd/E62172_01/html/E62184/gopyo.html#scrolltoc

http://docs.oracle.com/cd/E62159_01/html/E62171/gopqe.html#scrolltoc

Attachments

This solution has no attachment