New LUN Mapped from EMC Storage to Solaris Host seen by Format as Drive Not Available - Powerpath Shows all Paths as Unknown and Dead

Asset ID:	1-72-1946462.1
Update Date:	2018-03-07
Keywords:

Solution Type Problem Resolution Sure

Solution 1946462.1 : New LUN Mapped from EMC Storage to Solaris Host seen by Format as Drive Not Available - Powerpath Shows all Paths as Unknown and Dead

Applies to:

SPARC T4-1 - Version Not Applicable and later
Qlogic FC HBA - Version Not Applicable and later
Emulex FC HBA - Version Not Applicable and later
Solaris Operating System - Version 8.0 and later
Information in this document applies to any platform.

Symptoms

Solaris 10 server has two Oracle FC HBAs, two ports connected to the SAN to access and EMC storage array,
several old LUNs mapped, but there is a problem with a new LUN 10 (Host LUN ID 10)
format presents it as "drive not available"

The FC HBA is as follows:

c6 = qlc1 (fp5) -> /devices/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:devctl
================================================================================
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 660113 0 5006016a46e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
1 680313 0 5006016346e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
2 680413 0 5006016c46e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
3 680513 0 5006016546e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
4 680613 0 5006016e46e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
5 660b13 0 21000024ff432cb5 20000024ff432cb5 0x1f (Unknown Type,Host Bus Adapter)

c10 = qlc3 (fp7) -> /devices/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:devctl
================================================================================
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 650113 0 5006016246e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
1 670313 0 5006016b46e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
2 670413 0 5006016446e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
3 670513 0 5006016d46e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
4 670613 0 5006016646e0XXXX 50060160c6e0XXXX 0x0 (Disk device)
5 650b13 0 21000024ff3ab01f 20000024ff3ab01f 0x1f (Unknown Type,Host Bus Adapter)

On format output the LUN 0xa (10) is seen as "drive not available" on all paths and also under power path

     12. c6t5006016A46E0XXXXd10 <drive not available>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a46e0XXXX,a
     27. c6t5006016C46E0XXXXd10 <drive not available>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016c46e0XXXX,a
     42. c6t5006016E46E0XXXXd10 <drive not available>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016e46e0XXXX,a
     67. c6t5006016346E0XXXXd10 <drive not available>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016346e0XXXX,a
     68. c6t5006016546E0XXXXd10 <drive not available>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016546e0XXXX,a

     87. c10t5006016B46E0XXXXd10 <drive not available>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016b46e0XXXX,a
    102. c10t5006016D46E0XXXXd10 <drive not available>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016d46e0XXXX,a
    137. c10t5006016446E0XXXXd10 <drive not available>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016446e0XXXX,a
    138. c10t5006016646E0XXXXd10 <drive not available>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016646e0XXXX,a
    139. c10t5006016246E0XXXXd10 <drive not available>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016246e0XXXX,a

    153. emcpower1g <drive not available>
         /pseudo/emcp@1

While the other LUNs have no issue, example of LUN 0 :

      2. c6t5006016A46E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a46e0XXXX,0
     17. c6t5006016C46E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016c46e0XXXX,0
     32. c6t5006016E46E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016e46e0XXXX,0
     47. c6t5006016346E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016346e0XXXX,0
     48. c6t5006016546E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016546e0XXXX,0

     77. c10t5006016B46E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016b46e0XXXX,0
     92. c10t5006016D46E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016d46e0XXXX,0
    107. c10t5006016446E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016446e0XXXX,0
    108. c10t5006016646E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016646e0XXXX,0
    109. c10t5006016246E0XXXXd0 <DGC-RAID5-0219 cyl 53246 alt 2 hd 256 sec 40>
         /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016246e0XXXX,0

    152. emcpower0c <DGC-VRAID-0430 cyl 53246 alt 2 hd 256 sec 40>
         /pseudo/emcp@0

Notice that power path present all paths as unknown / dead for this disk:

Pseudo name=emcpower1a
CLARiiON ID=CKM0010450XXXX [RIMS_VM_DR]
Logical device ID=6006016066702A0048DB177528B0E311 []
state=alive; policy=CLAROpt; queued-IOs=0
Owner: default=Unknown, current=Unknown Array failover mode: 1
==============================================================================
--------------- Host --------------- - Stor - -- I/O Path -- -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3077 UNKNOWN unknown SP A5 active dead 0 0
3079 UNKNOWN unknown SP A2 active dead 0 0
3079 UNKNOWN unknown SP A4 active dead 0 0
3077 UNKNOWN unknown SP A3 active dead 0 0
3079 UNKNOWN unknown SP A6 active dead 0 0
3077 UNKNOWN unknown SP B4 active dead 0 0
3079 UNKNOWN unknown SP B5 active dead 0 0
3077 UNKNOWN unknown SP B2 active dead 0 0
3079 UNKNOWN unknown SP B3 active dead 0 0
3077 UNKNOWN unknown SP B6 active dead 0 0

Customer says we should see something like this, with a different logical device ID :

CLARiiON ID=CKM0010450XXXX [RIMS_VM_DR]
Logical device ID=6006016066702A0034047BECC8F8E011 [Pool3 - LUN304 - RIMS DR - DB1]

By other hand, the LUN 0xa (10) is configured and present on solaris (under cfgadm command)

bash-3.2$ grep ,10 cfgadm-al-o_show_FCP_dev.out
c6::5006016346e0XXXX,10 disk connected configured unknown
c6::5006016546e0XXXX,10 disk connected configured unknown
c6::5006016a46e0XXXX,10 disk connected configured unknown
c6::5006016c46e0XXXX,10 disk connected configured unknown
c6::5006016e46e0XXXX,10 disk connected configured unknown
c10::5006016246e0XXXX,10 disk connected configured unknown
c10::5006016446e0XXXX,10 disk connected configured unknown
c10::5006016646e0XXXX,10 disk connected configured unknown
c10::5006016b46e0XXXX,10 disk connected configured unknown
c10::5006016d46e0XXXX,10 disk connected configured unknown

On the last server boot, the ssd instances recognized LUN 0xa = 10 with no issue, all are online:

Nov 15 09:46:01 server01 scsi: [ID 583861 kern.notice] ssd105 at fp5: unit-address w5006016546e0XXXX,a: 680513
Nov 15 09:46:01 server01 genunix: [ID 936769 kern.notice] ssd105 is /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016546e0XXXX,a
Nov 15 09:46:01 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016546e0XXXX,a (ssd105) online
Nov 15 09:46:03 server01 scsi: [ID 583861 kern.notice] ssd108 at fp7: unit-address w5006016d46e0XXXX,a: 670513
Nov 15 09:46:03 server01 genunix: [ID 936769 kern.notice] ssd108 is /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016d46e0XXXX,a
Nov 15 09:46:03 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016d46e0XXXX,a (ssd108) online
Nov 15 09:46:07 server01 scsi: [ID 583861 kern.notice] ssd102 at fp5: unit-address w5006016c46e0XXXX,a: 680413
Nov 15 09:46:08 server01 genunix: [ID 936769 kern.notice] ssd102 is /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016c46e0XXXX,a
Nov 15 09:46:08 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016c46e0XXXX,a (ssd102) online
Nov 15 09:46:10 server01 scsi: [ID 583861 kern.notice] ssd109 at fp7: unit-address w5006016646e0XXXX,a: 670613
Nov 15 09:46:10 server01 genunix: [ID 936769 kern.notice] ssd109 is /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016646e0XXXX,a
Nov 15 09:46:10 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016646e0XXXX,a (ssd109) online
Nov 15 09:46:16 server01 scsi: [ID 583861 kern.notice] ssd103 at fp5: unit-address w5006016e46e0XXXX,a: 680613
Nov 15 09:46:16 server01 genunix: [ID 936769 kern.notice] ssd103 is /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016e46e0XXXX,a
Nov 15 09:46:16 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016e46e0XXXX,a (ssd103) online
Nov 15 09:46:16 server01 scsi: [ID 583861 kern.notice] ssd101 at fp7: unit-address w5006016446e0XXXX,a: 670413
Nov 15 09:46:16 server01 genunix: [ID 936769 kern.notice] ssd101 is /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016446e0XXXX,a
Nov 15 09:46:17 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016446e0XXXX,a (ssd101) online
Nov 15 09:46:23 server01 scsi: [ID 583861 kern.notice] ssd106 at fp7: unit-address w5006016b46e0XXXX,a: 670313
Nov 15 09:46:23 server01 genunix: [ID 936769 kern.notice] ssd106 is /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016b46e0XXXX,a
Nov 15 09:46:23 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016b46e0XXXX,a (ssd106) online
Nov 15 09:46:24 server01 scsi: [ID 583861 kern.notice] ssd104 at fp5: unit-address w5006016a46e0XXXX,a: 660113
Nov 15 09:46:24 server01 genunix: [ID 936769 kern.notice] ssd104 is /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a46e0XXXX,a
Nov 15 09:46:25 server01 genunix: [ID 408114 kern.notice] /pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a46e0XXXX,a (ssd104) online

BUT emc powerpath has some problems to identify the volume "6006016066702A0048DB177528B0E311"

Nov 15 09:47:30 server01 pseudo: [ID 129642 kern.notice] pseudo-device: emcp1
Nov 15 09:47:30 server01 genunix: [ID 936769 kern.notice] emcp1 is /pseudo/emcp@1
Nov 15 09:49:19 server01 emcp: [ID 801593 kern.notice] Info: Assigned volume 6006016066702A0048DB177528B0E311 to SPB
Nov 15 09:51:54 server01 emcp: [ID 801593 kern.notice] Info: PowerPath Auto Host Registration on CLARiiON-CKM0010450XXXX is unavailable: incompatible initiator information received from the array

Changes

New LUN mapped to the Solaris server from EMC storage-

Cause

Power path issue, it was identified a stale entry in powerpath (we don't have more details)

Solution

Check with EMC if the following action plan can solve the problem.

In this particular situation (problem described above) this solved the problem:

# powermt check --> cleared out all dead paths
# powermt save
# powercf -q

Then reboot with reconfiguration: reboot -- -r

After reboot, emc powerpath picked up the LUN, and format presented the LUN correctly.

Note. In case cfgadm command fails to configure EMC controllers, make sure a LUN 0 is mapped on the EMC storage controller port, see also:
Logical Unit Number(LUN) 0, a SCSI Requirement (Doc ID 1017707.1)
Solaris Storage Driver Best Practice for EMC Array Special Logical Device (LUNZ, VMCDB, Gatekeeper, etc ...) (Doc ID 1006505.1)

References

<NOTE:1005475.1> - What is the meaning of the message: "drive not available: formatting" from the Solaris [TM] format(1m) utility?

Attachments

This solution has no attachment