Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

In this Document

Goal

Solution

HOW TO CREATE A TOPOLOGY MAP

CONTROLLER CONNECTIONS

ENCLOSURE CONNECTIONS

SPOTTING TOPOLOGY ERRORS

LOGS

References

Applies to:

Goal

 To explain how to gather and understand the fscli command's topology option.  This document will not cover other fscli commands or any kind of troubleshooting getting this command to run.  Please consult the FSCLI Command Reference Guide or contact Oracle Support for assistance in those areas.

Solution

HOW TO CREATE A TOPOLOGY MAP

Log in to the FS1:

Using the command line (fscli), executing this command will generate a topology map:

INTERNAL: This information is also available in a log bundle in the active Controller directory in a file called PITMAN_MARKSDB.txt

It will produce an ASCII text file of the topology between the FS1-2 Controllers and Drive Enclosures (DEs) that if properly understood will be of great help in resolving any topology issues a system may encounter.  But to do so means to know the following details:

• Each Controller can have up to 3 SAS HBAs in them.  The first HBA (HBA 0) is always in slot 3, the second (HBA 1) is always in slot 6 and the last (HBA 2) is always in slot 2.
• Each SAS HBA has four ports:
• Interconnections between the Controllers themselves are always on SAS HBAS Ports 2 & 3.
• Connections between the Controllers and DEs are always on SAS HBA Ports 0 & 1.
• Each pair of SAS HBAs (from the same slot, but different Controllers) can support up to 10 DEs, 5 on each pair of ports.
• Connections from Controller 1 must be only to IOM 0, and Controller 2 must only connect with IOM 1.
• A DE is said to be downstream of another DE if it's IO Module (IOM) is further away from a Controller than another DE's IOM.  Conversely it is said to be upstream if it's IOM is closer to the Controller than another DE's IOM.
• Downstream connections are always on Port 0, upstream connections are on either Port 1 or Port 2.
  
  
  NOTE: Because a string of DEs has each end connected to the other Controller, whether or not a DE is downstream of another DE depends on the path or Controller you using.
   

CONTROLLER CONNECTIONS

The first section of the command output deals with the connections on each Controller (only Controller 1 shown for brevity):

It is best to view this data as a hierarchical representation with each Controller being at the top.  After the obligatory date/time stamp, the Controller is identified followed by it's WWN.  Next is the number of the SAS HBAs installed followed by their locations.  Lastly the port of that HBA is listed along with the object on the other end of the SAS connection.  

In the above example, all four ports are in use from each of the three HBAs.  Remember that ports 2 & 3 are inter-Controller connections and used for redundancy and failover.  Ports 0 & 1 are the connections to the first DE in that particular DE string.
 

ENCLOSURE CONNECTIONS

The second section of the topology map shows the connections of the Drive Enclosures (again, for brevity, only 3 DEs are shown):

 Again, viewing the data in a hierarchical light, each Drive Enclosure's name and WWN is listed followed by it's two IOMs (and their WWNs) and the 3 ports for each IOM.  And finally, what the object that IOM port is connected at the other end of this SAS cable.  It is now possible to trace these connections from the beginning of the string to the end.

Viewed from the IOM 0 side:

• Enclosure-01, IOM 0, port 0 connects (upstream) to Controller-01 HBA 0, port 0.
• Enclosure-01, IOM 0, port 1 connects (downstream) to Enclosure-02's IOM 0, port 0. 
• Enclosure-02, IOM 0, port 0 connects (upstream) to Enclosure-01 IOM 0, port 1.
• Enclosure-02, IOM 0, port 1 connects (downstream) to Enclosure-03's IOM 0, port 0. 
• Enclosure-03, IOM 0, port 0 connects (upstream) to Enclosure-02 IOM 0, port 1.
• Enclosure-03, IOM 0, port 1 is disconnected indicating the end of the string.
  
  Or,  Controller-01 ---> IOM 0 Enclosure-01 IOM1 --->IOM 0 Enclosure-02 IOM 1 ---> IOM 0 Enclosure-03

Viewed from the IOM 1 side:

• Enclosure-01, IOM 1, port 0 connects (upstream) to Enclosure-02 IOM 1, port 1.
• Enclsoure-01, IOM 1, port 1 is disconnected indicating the end of the string. 
• Enclosure-02, IOM 1, port 0 connects (upstream) to Enclosure-03 IOM 1, port 1.
• Enclosure-02, IOM 1, port 1 connects (downstream) to Enclosure-01's IOM 1, port 0. 
• Enclosure-03, IOM 1, port 0 connects (upstream) to Controller 2 on HBA 0 port 0.
• Enclosure-03, IOM 1, port 1 connects (downstream) to Enclosure-02's IOM 1, port 0.
  
  Or, Controller-02 ---> IOM 0 Enclosure-03 IOM 1---> IOM 0 Enclosure-02 IOM 1---> IOM 0 Enclosure-01

 SPOTTING TOPOLOGY ERRORS

 There are two items to look for that will indicate an obvious problem.

A Drive Enclosure that has all 3 ports disconnected:

A Drive Enclosure that is using the wrong IOM:

Note:  Port 0 of an IOM should always be in CONNECTED state as it is on the upstream path to the CONTROLLER.

LOGS

After the topology information will come a log of recent topology operations (Opcodes).  Some Opcodes are benign others can help identify intermittent topology issues.  All entries will start with a date and command followed by an Entry Class, Action Type and the Opcode.  Depending on what that Opcode is dictates what, if any, information will follow.  Below are a couple of examples of the entries that you may see that are not actionable:

 Here are a couple of examples that doe need attention:

Reviewing an extracted  log bundle for path troubleshooting.

command: FSInfo.pl -e
file: events_summary.txt
file: slammer/[MASTER_CONTROLLER]/PITMAN_MARKSDB.txt

The following examples use the topology of CONTROLLER-01 <>  ENCLOSURE-01 <> ENCLOSURE-02

First example:
Unplugging the SAS cable between Controller 1 Slot 3 port 0 and Drive Enclosure 1.  Notice that TWO Drive Enclosures are affected.

FSInfo -e:

events_summary.txt   get a RAID_EVENT_PARTIAL_OFFLINE_STATE_CHANGE for each affected drive (two affected enclosures)

PITMAN_MARKSDB.txt

Note that the output is how the > Controller < sees the SAS world.
Port 1 confusingly shows DISCONNECTED although the cable on Port 1 has not been removed, just the Port 0 cable.



Following reconnection.

cable exrercise #2
Unplug the SAS cable that goes between the following two points
/ENCLOSURE-01: IOM 0: Port 1 <==  ==> /ENCLOSURE-02: IOM 0: Port 0


FSInfo -e   here we see just ENCLOSURE-02 affected.

events_summary.txt

PITMAN_MARKSDB.txt

Following reconnection.

This solution has no attachment