A Rebooted IO Domain Reports: "WARNING: fp0: online

Asset ID:	1-72-1946680.1
Update Date:	2014-11-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1946680.1 : A Rebooted IO Domain Reports: "WARNING: fp0: online_driver fp failed"

Applies to:

SPARC T5-2 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Customer's Note:
Found warning messages for fp0 and fp2 when the I/O domain is booted up.

I tested moving pci_2 and pci_3 back to Primary but no similar warning messages was observed on Primary domain upon reboot. The messages only happened on I/O domain.

Analysis summary:
fp0 and fp2 are bogus and does not exist.

Detail analysis:
From the explorer data:

$ grep fctl: messages
Nov 5 09:30:47 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 09:30:50 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 09:36:45 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 09:36:48 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 09:40:31 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 09:40:34 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 12:09:41 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 12:09:41 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 14:28:55 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 14:28:55 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 15:55:03 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 15:55:03 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed
Nov 5 16:27:37 <server> fctl: [ID 290952 kern.warning] WARNING: fp0: online_driver fp failed
Nov 5 16:27:37 <server> fctl: [ID 290952 kern.warning] WARNING: fp2: online_driver fp failed

But these are the actual HBAs installed:
From the boot messages:

fp10 is /pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0/fp@0,0
fp4 is /pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0/fp@0,0
fp6 is /pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0,1/fp@0,0
fp8 is /pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0,1/fp@0,0
qlc0 is /pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0
qlc1 is /pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0,1
qlc2 is /pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0
qlc3 is /pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0,1

Which are also confirmed in /etc/path_to_inst:

"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0" 0 "qlc"
"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0/fp@0,0" 10 "fp"
"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0/fp@1,0" 11 "fp"
"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0,1" 1 "qlc"
"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0,1/fp@0,0" 8 "fp"
"/pci@380/pci@1/pci@0/pci@5/SUNW,qlc@0,1/fp@1,0" 9 "fp"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0" 2 "qlc"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0/fp@0,0" 4 "fp"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0/fp@1,0" 5 "fp"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0,1" 3 "qlc"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0,1/fp@0,0" 6 "fp"
"/pci@380/pci@1/pci@0/pci@6/SUNW,qlc@0,1/fp@1,0" 7 "fp"

The error most probably came from these symbolic links in /dev/fc:
But from experience this link is never really used at all.
Notice that fp0 is now fp10 and fp2 is now fp4

lrwxrwxrwx 1 root root 64 Mar 10 2014 fp0 -> ../../devices/pci@380/pci@1/pci@0/pci@5/
SUNW,qlc@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 66 Mar 10 2014 fp1 -> ../../devices/pci@380/pci@1/pci@0/pci@5/
SUNW,qlc@0,1/fp@0,0:devctl
lrwxrwxrwx 1 root root 64 Mar 10 2014 fp2 -> ../../devices/pci@380/pci@1/pci@0/pci@6/
SUNW,qlc@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 66 Mar 10 2014 fp3 -> ../../devices/pci@380/pci@1/pci@0/pci@6/
SUNW,qlc@0,1/fp@0,0:devctl
lrwxrwxrwx 1 root root 64 Mar 10 2014 fp4 -> ../../devices/pci@380/pci@1/pci@0/pci@5/
SUNW,qlc@0/fp@1,0:devctl
lrwxrwxrwx 1 root root 64 Mar 10 2014 fp5 -> ../../devices/pci@380/pci@1/pci@0/pci@6/
SUNW,qlc@0/fp@1,0:devctl
lrwxrwxrwx 1 root root 64 May 30 14:56 fp6 -> ../../devices/pci@380/pci@1/pci@0/pci@6/
SUNW,qlc@0/fp@2,0:devctl

Cause

Only happens in IO domain , not on primary domain.

Solution

The following has been tried but DID NOT fix the issue:

1. Clean up the device tree:
# devfsdadm -Cv

2. Totally recreating a new device tree by doing reconfiguration reboot after deleting /dev/fc, /devices/*, /dev/* and /etc/path_to_inst*

3. Updating to: Oracle Solaris 11.2.3.5.0

The Solution:

Connect the affected ports to the fabric and do:
# luxadm -e forcelip <path>

Attachments

This solution has no attachment