After OS Disk Replacement on Oracle Big Data Appliance bdachecksw/bdacheckhw Commands Fail with 'Wrong slot mapping to HBA target' Error

Asset ID:	1-72-1569762.1
Update Date:	2016-10-11
Keywords:

Solution Type Problem Resolution Sure

Solution 1569762.1 : After OS Disk Replacement on Oracle Big Data Appliance bdachecksw/bdacheckhw Commands Fail with 'Wrong slot mapping to HBA target' Error

Applies to:

Big Data Appliance Hardware - Version All Versions and later
Linux x86-64

Symptoms

After OS Disk replacement and configuration bdacheckhw/bdachecksw commands fail with below errors.

ERROR: Wrong slot 1 mapping to HBA target : 0
INFO: Expected slot 1 mapping to HBA target : 1
ERROR: Wrong slot 0 mapping to HBA target : 1
INFO: Expected slot 0 mapping to HBA target : 0

ERROR: Wrong mounted partitions : /dev/md0 /boot ext3 /dev/md2 / ext3 /dev/sd4 /u02 ext4 /dev/sd4 /u01 ext4 /dev/sd1 /u03 ext4 /dev/sd1 /u04 ext4 /dev/sd1 /u05 ext4 /dev/sd1 /u06 ext4 /dev/sd1 /u07 ext4 /dev/sd1 /u08 ext4 /dev/sd1 /u09 ext4 /dev/sd1 /u10 ext4 /dev/sd1 /u11 ext4 /dev/sd1 /u12 ext4
INFO: Expected mounted partitions : 12 data partitions, /boot and /
WARNING: Wrong matching label and slot : symbolic link to `../../sda4'
INFO: Expected matching label and slot : symbolic link to `../../sdb4'
WARNING: Wrong matching label and slot : symbolic link to `../../sdb4'
INFO: Expected matching label and slot : symbolic link to `../../sda4'

Output of 'mount -l' command shows that /dev/sda and /dev/sdb are mapped to wrong partitions

/dev/sdb4 on /u01 type ext4 (rw,nodev,noatime)
/dev/sda4 on /u02 type ext4 (rw,nodev,noatime)
/dev/sdc1 on /u03 type ext4 (rw,nodev,noatime)

The correct mapping is:

/dev/sda4 on /u01 type ext4 (rw,nodev,noatime)
/dev/sdb4 on /u02 type ext4 (rw,nodev,noatime)
/dev/sdc1 on /u03 type ext4 (rw,nodev,noatime)

Cause

The cause of this behavior is that the disks were configured in the wrong order during the OS Disk replacement and configuration step. In the case of configuring the disks in the wrong order the disk with highest slot number was configured first and then the disk with lowest slot number was configured.

When multiple disks are unconfigured, the disks need to be configured in the order from the lowest to the highest slot number.

Solution

To resolve the issue: 1) the node needs to booted into rescue image (by booting from usb ), 2) the disks need to be unconfigured and 3) then the disks need to be configured correctly.

Follow below steps to correctly configure the disks.

1) First determine the enclosure id. This information is required in subsequent steps.

> MegaCli64 pdlist a0 | grep "^Encl"

For example, assuming the enclosure id is 20, the output looks like:

Enclosure Device ID: 20
Enclosure position: 0
...

2) Boot the node into the rescue image.

Refer to 1570447.1 for steps on how to boot a BDA node into USB Rescue Mode using Grub Menu from Oracle Integrated Lights Out Manager (ILOM) interface Remote Console.

3) After reboot verify you have booted into USB Rescue Mode by checking the output from 'ls /'.

The 'ls /' output should show a file named: 'THIS_IS_THE_USB_DISK'.

4) Unconfigure the disks in order from the highest to the lowest slot number as below:

a) Take disk1 offline and unconfigure

# MegaCli64 PDOffline physdrv[20:1] a0

Adapter: 0: EnclId-20 SlotId-1 state changed to OffLine.

Exit Code: 0x00

# MegaCli64 CfgLdDel L1 a0

Adapter 0: Deleted Virtual Drive-1(target id-1)

Exit Code: 0x00

b) Check the status of the disks after unconfiguring disk1

# MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Slot Number: 0
Firmware state: Unconfigured(good), Spun Up
Foreign State: None
Slot Number: 1
Firmware state: Offline
Foreign State: None

c) Unconfigure disk0

# MegaCli64 CfgLdDel L0 a0

Adapter 0: Deleted Virtual Drive-0(target id-0)

Exit Code: 0x00

d) Check the status of the disks after unconfiguring both disk0 and disk1

# MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Slot Number: 0
Firmware state: Unconfigured(good), Spun Up
Foreign State: None
Slot Number: 1
Firmware state: Unconfigured(good), Spun Up
Foreign State: None

5) Correctly configure the disks in order from the lowest to the highest slot number as below:

a) Add disk0

# MegaCli64 CfgLdAdd r0[20:0] a0

Adapter 0: Created VD 0

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

Note:

If CfgLdAdd command fails because of cached data present, use MegaCli -DiscardPreservedCache -Ln a0 to clear the cache for the logical drive where n is the number of the slot used. For the above example it would be:

MegaCli64 -DiscardPreservedCache -L4 a0

Do NOT use CfgEachDskRaid0, this command reverses the slot and virtual disk numbers.

b) Check the state of the disks after adding disk0

# MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Slot Number: 0
Firmware state: Online, Spun Up
Foreign State: None
Slot Number: 1
Firmware state: Unconfigured(good), Spun Up
Foreign State: None

c) Add disk1

# MegaCli64 CfgLdAdd r0[20:1] a0

Adapter 0: Created VD 1

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

d) Check the state of the disks after adding both disks

# MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Slot Number: 0
Firmware state: Online, Spun Up
Foreign State: None
Slot Number: 1
Firmware state: Online, Spun Up
Foreign State: None
Slot Number: 2

6) Reboot the node after all the disks are 'Online, Spun Up'

References

<NOTE:1570447.1> - Steps to Boot Oracle Big Data Appliance Node into USB Rescue Mode using Grub Menu
<NOTE:1569754.1> - Oracle Big Data Appliance Node NOT reachable due to both OS Disks in Unconfigured Firmware State

Attachments

This solution has no attachment