SunFire V880 Panic on boot qlc@2 fail

Asset ID:	1-72-1900988.1
Update Date:	2017-04-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1900988.1 : SunFire V880 Panic on boot qlc@2 fail

Applies to:

Sun Fire V880 Server - Version All Versions to All Versions [Release All Releases]
Sun Fire V890 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Sun OS

Symptoms

1, System panics on boot showing panic[cpu6]/thread=180e000: vfs_mountroot: cannot mount root

2, Disk backplane failing during boot

WARNING: Device /pci@8,600000/SUNW,qlc@2 being marked with 'status' == fail

3, probe-scsi-all sees all internal disk

    ok probe-scsi - -all

    /pci@8,600000/SUNW,qlc@2
    LiD HA LUN --- Port WWN --- ----- Disk description -----
    0 0 0 500000e0106d5f11 FUJITSU MAP3735F SUN72G 1201
    1 1 0 500000e010777df1 FUJITSU MAP3735F SUN72G 1201
    2 2 0 500000e0106a5661 FUJITSU MAP3735F SUN72G 1201
    6 6 0 50800200001d2cf1 SUNW SUNWGS INT FCBPL9228
    3 3 0 500000e0106e4be1 FUJITSU MAP3735F SUN72G 1201
    4 4 0 500000e0106a6911 FUJITSU MAP3735F SUN72G 1201
    5 5 0 500000e0106f0321 FUJITSU MAP3735F SUN72G 1201
    8 8 0 2100000c50565823 SEAGATE ST373307FSUN72G 0307
    9 9 0 2100001862cc39a8 SEAGATE ST314655FSUN146G0691
    a a 0 2100001862caf6e7 SEAGATE ST314655FSUN146G0691
    b b 0 2100000c5056560c SEAGATE ST373307FSUN72G 0307
    c c 0 2100001862cc3775 SEAGATE ST314655FSUN146G0691
    d d 0 2100001862cfd57d SEAGATE ST314655FSUN146G0691

4, obdiag passes test 1

   >> Testing disk at loop ID: d
   Selftest at /pci@8,600000/SUNW,qlc@2 .................................. passed
   Pass:1 (of 1260) Errors:0 (of 0) Tests Failed:0 Elapsed Time: 0:0:8:29

no error found under obdiag test but the panic persist

Cause

Disk backplane fail qlc@2 path failed during booting

WARNING: Device /pci@8,600000/SUNW,qlc@2 being marked with 'status' == fail

Testing with obdiag all run fine

obdiag> test 1
Hit the spacebar to interrupt testing
Testing /pci@8,600000/SUNW,qlc@2
>> Testing RISC RAM (this may take a while)..........
>> Firmware copied
>> Waiting for loop to come up.
>> Waiting for firmware ready state
>> FCAL device count = 0xe
>> Found device with loop ID 0x7d (AL_PA = 0x1 )
>> Found device with loop ID 0x0 (AL_PA = 0xef )
>> Found device with loop ID 0x1 (AL_PA = 0xe8 )
>> Found device with loop ID 0x2 (AL_PA = 0xe4 )
>> Found device with loop ID 0x6 (AL_PA = 0xdc )
>> Found device with loop ID 0x3 (AL_PA = 0xe2 )
>> Found device with loop ID 0x4 (AL_PA = 0xe1 )
>> Found device with loop ID 0x5 (AL_PA = 0xe0 )
>> Found device with loop ID 0x8 (AL_PA = 0xd9 )
>> Found device with loop ID 0x9 (AL_PA = 0xd6 )
>> Found device with loop ID 0xa (AL_PA = 0xd5 )
>> Found device with loop ID 0xb (AL_PA = 0xd4 )
>> Found device with loop ID 0xc (AL_PA = 0xd3 )
>> Found device with loop ID 0xd (AL_PA = 0xd2 )
>> ISP2200 found at loop ID 0x7d
>> Enclosure services device found at loopid 0x6
>> Direct-access device ( disk 0 ) found at loop ID 0x0
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 0
>> Direct-access device ( disk 1 ) found at loop ID 0x1
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 1
>> Direct-access device ( disk 2 ) found at loop ID 0x2
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 2
>> Direct-access device ( disk 3 ) found at loop ID 0x3
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 3
>> Direct-access device ( disk 4 ) found at loop ID 0x4
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 4
>> Direct-access device ( disk 5 ) found at loop ID 0x5
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 5
>> Direct-access device ( disk 6 ) found at loop ID 0x8
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 8
>> Direct-access device ( disk 7 ) found at loop ID 0x9
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: 9
>> Direct-access device ( disk 8 ) found at loop ID 0xa
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: a
>> Direct-access device ( disk 9 ) found at loop ID 0xb
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: b
>> Direct-access device ( disk 10 ) found at loop ID 0xc
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: c
>> Direct-access device ( disk 11 ) found at loop ID 0xd
>> Waiting for disk to spin up (timeout in one minute)... Disk spun up.
>> Disk media testing - this will take a while.
>> Testing disk at loop ID: d
Selftest at /pci@8,600000/SUNW,qlc@2 .................................. passed
Pass:1 (of 1260) Errors:0 (of 0) Tests Failed:0 Elapsed Time: 0:0:8:29

System still panic during boot

panic[cpu6]/thread=180e000: vfs_mountroot: cannot mount root

000000000180b950 genunix:vfs_mountroot+370 (1861800, 188c800, 0, 129a800, 1865800, 6)
%l0-3: 0000030002866e00 0000000001858d90 000000000113d000 00000000018be400
%l4-7: 0000000000000600 0000000000000200 0000000000000800 0000000000000200
000000000180ba10 genunix:main+10c (18b2000, 180c000, 183b2c0, 10aa400, 0, 183c190)
%l0-3: 0000000000000001 0000000070002000 0000000070002000 0000000000000000
%l4-7: 0000000001841800 0000000000000000 0000000001815400 0000000001815648

To confirm that a single disk is not causing the backplane to be faulted. All internal disk be should pull out and tests run with different disks installed as a way to eliminate a single disk problem. Process of elimination is the only way to determine the correct cause of fault.

Solution

Replace disk BackPlane or disk as appropriate.

References

<NOTE:1008827.1> - Using obdiag to troubleshoot internal drives on the Sun Fire(TM)V880 server.
<NOTE:1523775.1> - V880 Crashed due to a disk failure

Attachments

This solution has no attachment