Pillar Axiom 300 Storage System
 
Pillar Axiom 500 Storage System
 
Pillar Axiom 600 Storage System
 

PLA-Support>Sun Systems>DISK>Axiom>SN-DK: Ax600
 

iSCSI Qualified Names are supposed to be world-wide unique identifiers.    When Solaris systems are prepared from common source or cloned drives, the iSCSI Qualified Name is also duplicated if copying of the /etc/iscsi/iscsi_v1.dbc is not suppressed.

In this Document

Symptoms

Cause

Solution

Created from <SR 3-6552931151>

Applies to:

Symptoms

A Solaris host running the Axiom Path Manager may show as Not Communicating in the Axiom GUI.  If any of the following happen, the hosts that are affected can change:

• The Axiom system is restarted
• The Active Pilot CU on the Axiom is restarted or changed
• The Solaris hosts are restarted
• The ethernet network between the Solaris hosts and the Axiom is interrupted, then restored.  

The "Not Communicating" status can have other causes.  This article documents how to identify and resolve the issue when this failure is due to duplicate iSCSI Qualified Names in multiple Solaris hosts.

This Not Communicating will happen any time there are multiple Solaris hosts using the same iSCSI Qualified Name, whether or not the host is using iSCSI.  

Only a single host with the same iqn will be allowed to log in.  The rest will fail the APM login due to duplicate iqn and be in "Not Communicating" status.   If the host that successfully logs in loses the TCP connection for any reason, some other Solaris host with the same iqn may log in successfully, which can result in the Communicating/Not Communicating status changing. 

Cause

The host status is displayed as Not Communicating if that host does not currently have an APM daemon login session to the Axiom Pilot, but has logged in at some point in the past.  

The three statuses are:

• Not Registered   No APM login has ever happened for a host that has been manually associated.
• Not Communicating   The APM host has logged in before, but is currently not logged in.
• Communicating    The APM host currently has a connection to the Axiom Pilot.

These error messages are due to two or more Solaris hosts having the same iSCSI Qualified Name for their initiator.  Even if the hosts do not use iSCSI this can still be an issue and must be fixed.   The APM daemon running on Solaris gets the iSCSI initiator-node name from the the iSCSI Management API library (/usr/lib/libima.so).

Solution

 The solution is to cofigure each Solaris host with a unique iSCSI initiator node name. 

There are two methods to verify this issue. 

• Generate  APM debug logs at the Solaris Host and examine the Daemon Trace.
• Check the iSCSI Initiator identity with iscsiadm

Checking the iSCSI Qualified Name, or iqn

To check the configured initiator node name on a Solaris host, run the following command and look at the Initiator node name field: For more information on the iscsiadm command, see iscsiadm(1M)

Collecting APM Host Debug Logs and verifying log entries

Since the APM host is not communicating with the Axiom pilot, debug logs must be generated and collected at the host.  To create debug logs, type /opt/pillar/sbin/axiompmd -d. This will produce a logset directory in /var/log/axiompmd, named for the time of log creation.  For example: Tue_Dec_18_14_20_24_2012/. The Daemon_Trace is in that directory, with the hostname prefixed. For example, if the hostname is shost3, the file name will be shost3_DaemonTrace.txt. This file can be viewed directly on the Solaris host.  The typical sequence in the axiompmd debug daemon trace log is:  some lines removed

date/time INF 00000004 PrtIscsiInfo: Initiator Name  iqn.1986-03.com.sun:01:0010e0214064.50bf35d7
date/time INF 00000004 PrtIscsiInfo: Number of iSCSI Adapters 0  Solaris may use a Software Initiator.  If the software initiator is used, the number of iSCSI specific adapters will be "0"
INF 00000004 MccNotePilot: recording new pilot 10.10.10.10 The IP address of the Axiom Pilot
pilot_connect: Connected to Axiom 10.10.10.10 from Host 10.10.10.13
ssl_sendMsg: Sent XML Message
ssl_recvMsg: Received XML Message
pilot_connect: Logged in to Axiom 10.10.10.10, Session ID A873DAC64155945725E155DCCB122364

If you see the above Connected to and Logged in messages, the communications path from the APM host and the Axiom are available.   Then the APM host and Axiom exchange APM version negotiation.  The current supported version is "2"

MccSendRequest: Queue Request MGMTILIB_NEGOTIATE_VERSION to 10.10.10.10
MccHandleResponse: Received Good Response to MGMTILIB_NEGOTIATE_VERSION from 10.10.10.10
MccHandleResponse: Message versions: Pilot 10.10.10.10 is 2, Daemon is 2, Negotiated 2

The APM daemon sends the Configure San Host request. This request includes the hostname, a list of all host HBAs and HBA WWNs discovered, all iSCSI initiator identities (whether or not iSCSI is in use), and the LUNs and paths that queries to the Host multipath driver API can detect.

MccSendRequest: Queue Request MGMTILIB_PERFORM_CONFIGURE_SAN_HOST_V2 to 172.32.221.19

If this configure request is accepted, the Axiom will return the load balancing setting from the Axiom, a list of all LUNs configured to any of the host's WWNs or Initiators, any LUNs configured to the hostname or host group, and any globally mapped LUNs. The Axiom will fail the configure request if it contains any duplicates in host name, host WWN, or iqn. The following is a failed login, where the Status 3507 indicates that some other host is using one or more of the initiators.

MccHandleResponse: Received Response to MGMTILIB_PERFORM_CONFIGURE_SAN_HOST_V2 from 172.32.221.19 with Status 3507
MccHandleResponse: Resetting Connection to 172.32.221.19 after Error in Response
pilot_disconnect: Disconnecting from 172.32.221.19

Modifying the iSCSI Qualified Name, or iqn

Following is an example to disable APM, modify the initiator node name via  the 'iscsiadm modify initiator-node --node-name <initiator-node name (should be globally unique)>' command, and re-enable APM:

Verify changes to the initiator name took place:

 At this point APM should be able to log into the Axiom pilot and the host should report a Communicating state in the GUI.

If this does not solve the issue, you may be asked to provide the output of iscsiadm list initiator-node and another set of APM debug logs to the Support Center for assistance.

This solution has no attachment