Sun Fire V880z Visualization Server
 
Sun Fire V490 Server
 
Sun Fire V480 Server
 
Sun Fire V880 Server
 
Sun Fire V890 Server
 

PLA-Support>Sun Systems>SPARC>Workgroup Servers>SN-SPARC: SF-Vx80
 
_Old GCS Categories>Sun Microsystems>Servers>Entry-Level Servers
 

In this Document

Goal

Solution

Architecture

Background

Single Backplane (Dual Path Shown)

Dual Backplane (Dual Backplane Split Path Shown)

Finding the Error

Finding The Disk

Identify the Configuration

Find the Disk Slot

Volume Manager Considerations

For Further DPM Troubleshooting Steps if your issue is not solved by the other steps see the references section Below

References

Applies to:

Goal

<p >*Goal
<span >Enter the goal of the document. What does the customer want to accomplish?

Troubleshoot FCAL Disk issues on V480, V490, V880, V890

Solution

• Architecture

  • Background

Fibre Channel (FC) is a high-performance serial interconnect standard designed for bidirectional, point-to-point communication among servers, storage systems, workstations, switches, and hubs.

FC-AL devices employ a high-performance Gigabit serial interface, which supports multiple standard protocols such as Small Computer Systems Interface (SCSI) and Asynchronous Transfer Mode (ATM).

By supporting these standard protocols, FC-AL preserves any investment in existing legacy systems, firmware, applications, and software.

• • • Supports 100-Mbyte per second data transfer rate (200 Mbytes per second with dual porting)
    • Capable of addressing up to 127 devices per loop
    • Provides for reliability, availability, and serviceability (RAS) features such as hot-pluggable and dual-ported disks, redundant data paths, and multiple host connections
    • Supports standard protocols such as IP and SCSI
    • V480, V490 is a very simple implementation with 2 disks on a single Backplane.   The disks can be accessed by 2 paths either the internal controller or through an optional second path from a PCI card. 
    • Internal disks:
      • Disk 1       /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@WWN,0 (cXt0d0 lower disk)
        Disk 2       /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@WWN,0 (cXt1d0 upper disk)
      V880, V890 is a more complex configuration with a single or dual Disk Backplane ( Base Backplane and Expansion Backplane).
      The dual Backplanes can be joined or split, and each can have a single path or dual path.One path of the potentially 4 paths will be the internal controller:
      
      • Onboard FCAL-loopA    /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@WWN,0

  • Single Backplane (Dual Path Shown)

Detailed Wiring    Single Backplane Loop A         Single Backplane Loop  A  &  B

• • Dual Backplane (Dual Backplane Split Path Shown)

Detailed Wiring    Dual Backplane Joined Loop A          Dual Backplane Joined Loop    A  &  B

• • Finding the Error

1. Check the messages files to determine the paths that have the problem

 
2. Find the paths attached to the internal drives

 

You can have multiple paths to the internal drives such as /dev/es/ses0 and /dev/es/ses1. 
Several of options of the luxadm commands support running these commands to different ses devices.
For instance, ses0 is logically associated with FCloop. If you have another path to the internal drives you will have a different ses instance for that path. 
Counters are maintained on a per port.

 
3. Display the loop - check for general status information.  Login failed means that a drive is not present.

 

 
4. Dump the loop map of the loop in question.  This gives the position on the loop each device is on.  
Also in this case you can dump each instance of the ses devices by giving dump_map the ses instance.
(luxadm  -e dump_map /dev/es/ses0, luxadm -e dump_map /dev/es/ses1)

 
 
 
5. Dump the Link Error Status Block - This is where it gets tricky.  CRC errors are a good indication that there is a loop problem, usually from an upstream member.
Every counter needs to be accessed at in an incremental fashion since these are totals.  
It would be good to take a snapshot of the counters then take another snapshot after you see the problem you are experiencing.
Then compare the two.  CRC errors are indicators of a problem on the loop, but do not provide a reliable path to the bad hardware in all cases.  
High numbers of sync loss & invalid word don't necessarily mean there is a problem.
Some positions typically have higher "sync loss" than other ones.

 

NOTE: These LESB counts are not cleared by a reset, only power cycles.
These counts must be compared to previously read counts.

• • Finding The Disk

  • Identify the Configuration

The type of configuration can be identified in an explorer using various commands in the disk sub-directory in explorer.   ( explorer.838c4565.v880e-2012.01.17.21.45/disks ).  

The first step is to locate all the files that the erring disk is present in.   `grep -l <wwn>  *` in the disks sub-directory can be used.   WWN is the World Wide Number for the disk. 

Usually, the last 4 or 5 digits is enough to have a unique WWN

In the example above df73 shows up in several places.   It shows up in two different `luxadm -e dump_map` commands.  The dump_map looks at a device path and displays all FC devices on the path.

The disk WWN showing up in 2 dump_map files means the disk is dual pathed.   Notice that one of the paths is /pci@8,600000/SUNW,qlc@2/fp@0,0 which we know is the internal controller on the V880/V890. 

The other path is a PCI Card in Slot 8.The other way to see the number or paths is to look at the `luxadm display <enclosure>'.   In this case, the enclosure is 50800200001f6978 which is the node WWN. 

Each Backplane is treated as an individual enclosure in split Backplane configurations.  In Joined backplane configurations, the backplanes are treated as one enclosure.

Other common names for the enclosure/backplane on the V880/V890 are FCLoop and FCloop2.  FCloop2  is usually the name given to the Expansion Backplane in a split configuration. 

The name can be set with the luxadm command, so the enclosure name is not confirmation of a certain configuration.

Another way to determine the configuration type is by looking at the `luxadm display <enclosure>` output:

Notice in the output above because we are looking at FCloop2 we are most likely looking at the expansion Backplane. 

Notice further down in the output the various loops and Backplanes are listed.   #2 and #3 are the indicators for the Expansion Backplane. 

They both show a status of O.K.  The Base Backplane shows as not installed because the HBA that is accessing this enclosure cannot see the Base Backplane because the backplanes are Split.  

In this example we have a dual backplane and the backplanes are Joined, and the disks are controller by 2 controllers Loop A & B:

First thing to look at is all 12 disks are present.  This means the backplanes are joined.  Notice each backplane is connected to Loop A & B.  

#0, #1, #2, #3  show as O.K.  We also see near the bottom that Backplane  A  & B  are O.K.

This drawing has details of how this configuration is wired.   Dual backplane Loop    A  &  B

• • Find the Disk Slot

After the configuration type is determined to find which slot the disk is installed is the next step to replacing the disk. 

In all of the output above with 'luxadm display <enclosure>' the Disk SLOT number is the slot on the V880/V890. 

Often the 'luxadm display <enclosure>'  is missing from explorer, or not in the data sent from the customer.   The slot can also be determined from the `luxadm -e dump_map` command.

 In the above luxadm output the AL_PA of the Disk Node WWN can be compared to the AL_PA to Slot Chart to determine the disk Slot.  

For Example, Disk WWN 20000011c6f0cb40 has an AL_PA of d3.   AL_PA of d3 is associated with Disk Slot 10.

The disk target as it appears in format can be useful as well in some situations.  With a Dual Backplane Joined configuration the disks are numbered in the chassis:

Notice that the targets go from t0 to t5, then t8 to t13.   If format shows a similar target layout, it is a good assumption that the disks are V880/V890 Internal FC drives. 

With a Split configuration, it is not as simple to determine. The Expansion Backplane is seen as a second Enclosure, and the devices will target starting at t0 again:

• • Note on V480/V490:   The AL_PA for Disk 0 and Disk 1 are the same as for the V880/V890  ( EF & E8 ).   The Target for V480 and V490 are always the Disk slot Number.

  •  Volume Manager Considerations

 Solaris Volume Manager (SVM) and Veritas Volume Manager (VXVM) provide a means to control and utilize the two paths to the disk as one device. 

SVM uses MPXIO, while VXVM uses Dynamic Multi-Pathing (DMP).

Customers are strongly encouraged to use one of these technologies when using two paths to access an internal disk on V480/V490/V880/V890.

With both technologies, the Systems Administrator configures his application to access a meta-device which is aware of both paths to the disk and can load balance, and failover as required.

From a troubleshooting standpoint, this gives another source of errors that will need to be examined.  The errors will often call out the meta-device, the physical device or both. 

 The device in VXVM will look very similar to a normal Solaris cXtXdX number.  For example /dev/vx/dmp/c1t0d0.   The path contains a different basename than the usual Solaris disk. 

The path contains vx/dmp.   Veritas uses the first disk as the meta-device name so c1t0d0 is the equivalent to c1t0d0 with potentially another path to the disk. 

The `vxdisk list <disk>' command can be used to see both paths.

Near the bottom, it can be seen that there are 2 paths to this disk, one on c1 and one on c2.   The disks both show up in format as they would normally.   From format you can get the complete device path. 

The luxadm  command behaves as normal with VXVM and DMP.

Solaris Volume Manager behaves differently with respect to device paths when enabled.  SVM removes the underlying devices in /dev/dsk and leaves a single meta-device.  

This meta-device can be recognized by the scsi_vhci in the device path.

The other noticeable difference is the very long target number 20000000871D9B6A.  20000000871D9B6A is the Disk WWN. 

It should be noted that the hexadecimal numbers in the WWN are all converted to upper case to differentiate the d0 in the cXtXdX from being part of the WWN.  

That WWN can then be located in the various luxadm outputs to determine the physical disk location.

For additional DMP and MPXIO troubleshooting see the References section Below.

• Common Issues
  • # format" showing high target ID's assigned to drives & OBP Message - "WARNING: Internal FCAL selectable ID's have been reprogrammed with value 0  HELP Problem and Error Message:

After installing an Expansion BP the Solaris OS command "format" printed out the following (Showing high target ID's assigned to drives in the Expansion BP) and a "warning" message from OBP.

OBP Message:

TIP Solution/Workaround

The high target ID's can be a result of the loop(s) not getting initialized correctly. Clean up these devices under Solaris and issue another "boot -r".

If the high targets ID's persist and you're also seeing the above OBP "warning" message, chances are the Expansion BP is not getting identified correctly.

Reboot the system again. If on every reboot you see the OBP "warning" message then you most likely have a bad Expansion BP. Replace Expansion Backplane.

• • "Fast Data Access MMU Miss" during OBP/POST and no disk LED's are illuminated in Base BP (but are in Expansion BP).

HELP Problem and Error Message:

"Fast Data Access MMU Miss" during OBP/POST and also during "reset-all".

Checking for disks comes up empty with, "probe-scsi-all" and no disk LED's are illuminated in the Base BP (but are in the Expansion BP).

TIP Solution/Workaround

Since the Expansion BP is receiving power, but the Base BP isn't. There is a good chance the PDB is OK, but don't rule it out. We need to check voltages at P16 (which plugs into J0100 of the FCAL Base BP).

1. If all P16 voltages are present: Replace Base Backplane
2. If one or more P16 voltages are missing: Inspect/Reseat "PDU(P15) to Base BP(P16) cable (530-2841)" and recheck P16 voltages.
3. If one or more P16 voltages are missing: Check voltages at P15.
3a. If one or more P15 voltages are missing: Replace PDB
3b. If all P15 voltages are present: Replace cable (530-2841)

• • "Fast Data Access MMU Miss" during OBP/POST and no disk LED's are illuminated in the Base and Expansion BP's.

HELP Problem and Error Message:

"Fast Data Access MMU Miss" during OBP/POST and also during "reset-all". Checking for disks comes up empty with, "probe-scsi-all" and no disk LED's are illuminated in the Base and Expansion BP's.

TIP Solution/Workaround

Since disk LED's are not illuminated in both the base and expansion backplanes, we need to check power at P15 and P42 of the PDU:

1. If one or more P15/P42 voltages are missing: Replace PDB
2. If all P15/P42voltages are present, then Check voltages at P43 (Expansion BP): Note:If power isn't getting to the Expansion BP
(ie. Bad 530-2863 Cable), can cause both baseplane(Base and Expansion) to have no illuminated drive LED's.
3. If one or more voltages are missing at P43: Inspect/Reseat "PDU(P42) to Expansion BP(P43) cable (530-2863)" and recheck P43 voltages, but if same results: Replace cable (530-2863)
4. If all P43 voltages are present: Chances are one of your FCAL baseplanes causing the problem. Replace Expansion BP first. If still fails; Replace Base BP

• • No disk LED's illuminated in Expansion BP, "backplane 1" errors are present during "reset-all"...

HELP Problem and Error Message:

No visible errors during OBP/POST, no disk LED's illuminated in Expansion BP, "baseplane 1" errors are present during "reset-all", and no disks (just controllers) are seen during a "probe-scsi-all".

TIP Solution/Workaround

Inspect/Reseat "I/O Board (P18/J3805)) to Base BP (P19/J0800) cable (530-2840)" and retry, but if same results: Replace cable (530-2840)

• • No disk LED's illuminated in Expansion BP, "baseplane 0" and "Can't access LM75 device!" errors are present during "reset-all"

HELP Problem and Error Message:

No disk LED's eluminated in Expansion BP, "baseplane 0" and "Can't access LM75 device!" errors are present during "reset-all" & OBP/POST, and high target ID's are seen during a "probe-scsi-all".

IP Solution/Workaround

Inspect/Reseat I/O Board (P18/J3805) to Expansion BP (P20/J0800) cable (530-2840) and retry, but if same results: Replace cable (530-2840)

If problem persists replace the fcal baseplane after doing further troubleshooting steps to determine which baseplane is failing. (501-5993) Further DPM Troubleshooting Steps

• • "probe-scsi-all" is showing a "Loop ID (LiD)" of 7d (125 decimal, highest possible ID on the loop) ....

HELP Problem and Error Message:

"probe-scsi-all" is showing a "Loop ID (LiD)" of 7d (125 decimal, highest possible ID on the loop) and two "Hard Address (HA)" 6's. Neither is valid.

Error Output:

Good Output:

TIP Solution/Workaround

There is one cable that could cause this type of error and it's the FCAL Backplane ID cable, which is part of cable (530-2863) that goes between each FCAL Backplane.

Inspect/Reseat "Base BP (P1) to Expansion BP(P2) cable (530-2863)" and retry, but if same results: Replace cable (530-2863)

• •  " Loop is not up - no targets to test."   &  "ISP2200 reported that the loop is down" seen in OBDIAG.

HELP Problem and Error Message:

  
TIP Solution/Workaround:

A bad disk is preventing the loop from coming up.  Try removing all disks except one.  Test again.  If still seeing the error the disk you chose may be the bad one.  Try another.

If loop works with that one disk, use process of elimination to locate the faulty disk by adding disks and testing. It is also possible the controller or disk backplane is faulty and causing this error. 

• • No disks showing up under the internal FCAL controller (qlc@2):

HELP Problem and Error Message: No disks showing up under the internal FCAL controller (qlc@2):

TIP Solution/Workaround

Inspect/Reseat "Motherboard [(Loop A) A/FCAL(IN) & B/FCAL(OUT)] to Base BP [(Loop A) A/J0201 & B/J0200]" cable (530-2623)" and retry, but if same results: Replace cable (530-2623)

If an Expansion BP is present; cable (530-2621) could cause this problem too.

Oddly enough, if the link between "Base BP [(Loop A) E/J0500] & Expansion BP [(Loop A) B/J0200]" were to fail, you would also see the above displayed by "probe-scsi-all".

If the cable (530-2623) does not fix your problem and you have an Expansion BP. Inspect/Reseat cable (530-2621) and retry, but if same results: Replace cable (530-2621)

• • No disks showing up under the internal FCAL controller (qlc@2):

HELP Problem and Error Message:

Only Base BP disks showing up under the internal FCAL controller (qlc@2):

TIP Solution/Workaround

Inspect/Reseat "Base BP [(Loop A) E/J0500 & F/J0501] to Expansion BP [(Loop A) A/J0201 & B/J0200]" cable (530-2621)" and retry, but if same results: Replace cable (530-2621)

• • For Further DPM Troubleshooting Steps if your issue is not solved by the other steps see the references section Below

 KK

References

This solution has no attachment