Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-75-1401425.1
Update Date:2017-06-11
Keywords:

Solution Type  Troubleshooting Sure

Solution  1401425.1 :   Sun Storage 7000 Unified Storage System: iSCSI Troubleshooting - Cause Determination  


Related Items
  • Sun ZFS Storage 7320
  •  
  • Sun Storage 7210 Unified Storage System
  •  
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun ZFS Storage 7420
  •  
  • Sun Storage 7310 Unified Storage System
  •  
  • Sun Storage 7110 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
Related Categories
  • PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
  •  
  • _Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
  •  




In this Document
Purpose
Troubleshooting Steps
 Performance Problems
 Initiator Host Boot Problems
 Connectivity Problems
 Identifying if ZFSSA Is Presenting the LUN to the initiator host correctly:
 Identifying if ZFSSA Discovery Is Set up Correctly:
 Identifying ZFSSA Authentication Problems:
 iSCSI Network Connectivity Information
References


Applies to:

Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Purpose

This document helps you determine the cause of iSCSI problems on the Oracle ZFS Storage Appliance (ZFSSA), it will guides you through identifying causes for iSCSI connectivity problems.
It will also point to related documents for performance and boot issues. This document does not cover generic network issues.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Troubleshooting Steps

iSCSI problems falls into the following categories.

  • Performance Problems
  • initiator Host Boot Problems
  • Connectivity Problems

Performance Problems

Please refer to the respective product documentation for performance issues on the initiator side.  However,  if you have reason to believe the initiator performance is not the issue and the performance problem is caused by ZFSSA, refer to the following document.

Document: 1331769.1 Sun Storage 7000 Unified Storage System: How to troubleshoot performance issues.

Initiator Host Boot Problems

If you are experiencing problems booting initiator hosts from an iSCSI LUN on the ZFSSA, and your initiator host is running Oracle Solaris operating system, please refer to respective product documentation.  Currently iSCSI boot is supported only on Solaris 11.

If you are experiencing problems booting initiator hosts from an iSCSI LUN on the ZFSSA but the initiator hosts are running other operating systems, consult with vendor of the operating system first to rule out known problems with the operating system.

If the booting problem seems to be caused by loss of connection to iSCSI LUN, move on to the following section about iSCSI connectivity.

Also, please refer to the following documents if the client operating system is Oracle Solaris:

Document : 1340245.1 - zpools From iSCSI Will Not Mount After Reboot
Document : 1489871.1 - Solaris Volume Manager (SVM) Mirrored Root Disk Server/System/Node Can Not Boot. Troubleshooting. Resolution Path
Document : 1539410.1 - Using 'init 5' or 'init 6' on a System with UFS or ZFS on iSCSI LUNs may hang and report "ISCSI_DOOR_DAEMON_SYSLOG_PP[nnnnn]: [ID nnnnnn daemon.emerg] @ iSCSI initiator service exited with sessions left."


Connectivity Problems

iSCSI connectivity problems will reside in the following areas:

  • ZFSSA is not presenting the LUN to the initiator host
  • Initiator side problems
  • Authentication Problem
  • iSCSI related services or ports are not available
  • Underlying transport problems such as Infiniband fabric

Identifying if ZFSSA Is Presenting the LUN to the initiator host correctly:

Initiator hosts may have difficulties accessing to iSCSI LUNs on ZFSSA when one or more of the following criteria are met.

  • iSCSI service is not running on the ZFSSA correctly.
  • The LUN has not been created yet.
  • The LUN is not assigned to the correct host group or target group, therefore ZFSSA does not present LUN to the initiator IQN.
  • The configured initiator IQN is wrong.

In order to make sure ZFSSA is presenting iSCSI LUN correctly, check if the following criteria are met.Check initiator side IQN is defined correctly.

  • Check what initiator group the initiator IQN belongs to. See documents:

Document 1395907.1 Sun Storage 7000 Unified Storage System: How to identify the initiator IQN on ZFSSA.

Document 1395943.1 Sun Storage 7000 Unified Storage System: How to identify the initiator group that IQN belongs to on ZFSSA.

  • Check what target group the ZFSSA target IQN belongs to. See documents:

Document 1395320.1 Sun Storage 7000 Unified Storage System: How to identify target IQN on ZFSSA.

Document 1395356.1 Sun Storage 7000 Unified Storage System: How to identify the target group that the target IQN belongs to on the ZFSSA.

  • Check target is listening to the correct interface (IP or IPMP address) and make sure the LUN is correctly assigned.

Document 1395923.1 Sun Storage 7000 Unified Storage System: How to check LUN properties on ZFSSA.

Identifying if ZFSSA Discovery Is Set up Correctly:

There are three methods in iSCSI to discover targets.  They can be achieved by setting up one of the followings.

  • SendTargets – Target IP address
  • iSNS – iSNS server address
  • Static – Target IP address and target name

For configuration details, refer to "Configuring Solaris iSCSI Initiators" section of "Oracle Solaris 11.1 Administration: SAN Configuration and Multipathing" manual.
http://docs.oracle.com/cd/E26502_01/html/E29008/index.html

You can confirm what values are set by running the commands below on the initiator hosts.  (The below example is on Solaris.)

(1) Identifying what discovery method is enabled.

root@lab1# iscsiadm list discovery
Discovery:
        Static: disabled
        Send Targets: enabled
        iSNS: disabled

iscsiadm list discovery shows you what method of discovery is currently enabled. The above example has SendTargets enabled.

(2) Identifying discovery is correctly set.

(2)-1 Discovery address (for SendTargets)

In case you are using SendTargets, you need to check if discovery address is correctly set.

root@lab1:~# iscsiadm list discovery-address -v
Discovery Address: 192.168.0.1:3260
        Target name: iqn.1986-03.com.sun:02:0d57a6d4-59cd-e230-e138-c1556efdf71a
                Target address:  192.168.0.1, 2
        Target name: iqn.1986-03.com.sun:02:381a918b-9077-4ea0-91b3-8e96e5650673
                Target address:  192.168.0.1, 2

 The above example shows the discovery address is 192.168.0.1 and initiator has already found two different target IQNs.

(2)-2 Static configuration

root@lab1:~# iscsiadm list static-config
Static Configuration Target: iqn.1986-03.com.sun:02:0d57a6d4-59cd-e230-e138-c1556efdf71a,192.168.0.1:3260


The above example shows your static configuration is set to IQN of iqn.1986-03.com.sun:02:0d57a6d4-59cd-e230-e138-c1556efdf71a on the node with an IP address of 192.168.0.1 via port 3260.
(Port 3260 is standard port for iSCSI.)

(2)-3 iSNS

With iSNS, both ZFSSA and Solaris hosts need to register themselves to an iSNS server.  If they are registered correctly, IQNs must show up in iSNS server database.
In order to check each ZFSSA and Solaris hosts have correct iSNS settings, follow the below steps.

In case of iSNS, ZFSSA, configuration services iscsi show will let you know the IP address of the iSNS server currently set.

lab1:> configuration services iscsi show


Properties:
                      <status> = online
                          isns = true
                    isnsserver = 192.168.150.14:3205
                  radiussecret = (unset)
                  radiusserver = (unset)

The above example shows the iSNS server is set to 192.168.150.14 via port 3205. (3205 is a standard port for iSNS communication.)

 

In case of Solaris host, use the following commands.

root@oratkysn3:~# iscsiadm list isns-server
iSNS Server IP Address: 192.168.0.14:3205


root@oratkysn3:~# iscsiadm list isns-server -v
iSNS Server IP Address: 192.168.0.14:3205
    Target name: iqn.1986-03.com.sun:02:e19b6a6d-336c-471e-e973-f3bf4e741191
        Target address:  192.168.0.20:3260, 2

 
The first command shows iSNS server is set to 192.168.0.14 via port 3205.
The second one shows the iSNS server is set to the same IP address and port number and also telling us a target has been found on iSNS database with IQN of iqn.1986-03.com.sun:02:e19b6a6d-336c-471e-e973-f3bf4e741191, at the IP address of 192.168.0.20.  The second command does not work if iSNS discovery is disabled.

The targets found above only means initiators were able to find targets.  It requires iSCSI login to access to the LUN, thus we need to make sure the authentication is intact.  If the discovery is set correctly, please move on to "Identifying ZFSSA Authentication Problems" section.

 

**The below was the original text of this section.  I am thinking of eliminating Document 1398633.1. - 2013.03.02 Hisao Tsujimura

This section is valid ONLY WHEN you are using iSNS as a discovery method of iSCSI devices from the host.
Refer to the following document to verify if iSNS is set up correctly.
Document 1398633.1 Sun Storage 7000 Unified Storage System: How to check iSNS/RADIUS settings on ZFSSA.
Also, consult with the iSNS server manual to verify the ZFSSA target IQN is correctly registered.


Identifying ZFSSA Authentication Problems:

iSCSI initiators cannot communicate to the iSCSI targets when the initiator and the target do not agree on the authentication method, or authentication is not successful. The following document will help identify what authentication mechanism is used against the iSCSI targets.

There are three authentication methods.

  • None - no authentication, in this case the IQN of the host initiator must be defined manually on the ZFSSA.
  • CHAP (Challenge and Password) - a CHAP secret will need to be set
  • RADIUS - No password or secret needs to be set

None

If no authentication mechanism is used, the initiator IQN must be defined manually.
The following should be left blank.

  • CHAP Name and CHAP secret on initiator
  • CHAP Name and CHAP secret on target

RADIUS

If RADIUS authentication is used, the RADIUS properties must be set correctly in the iSCSI service screen.
Please refer to the following document. Check the IP address or host name of RADIUS server. If the host name is used, make sure it can be resolved to it's IP address correctly.

Document 1398633.1 Sun Storage 7000 Unified Storage System: How to check iSNS/RADIUS settings on ZFSSA.


2013.03.02 - Hisao Tsujimura

When I eliminate 1398633.1, I need to write out details on how to check RADIUS settings here.


CHAP

If CHAP is used check the following:

  • Check that initiator CHAP names and secrets match on both sides.
  • Make sure that the target CHAP name and secret do not match those of any of the initiators.

Refer to the following documents to identify your CHAP names
(You cannot view CHAP secrets thus you need to set them up again if you are unsure.)

Document 1401469.1 Sun Storage 7000 Unified Storage System: How to identify target CHAP name on ZFSSA .
Document 1401493.1 Sun Storage 7000 Unified Storage System: How to identify initiator CHAP name on ZFSSA.
Document 1181788.1 Sun Storage 7110 Unified Storage System: How to configure unidirectional CHAP for Solaris iSCSI initiator.
Document 1181843.1 Sun Storage 7110 Unified Storage System: How to configure bidirectional CHAP for Solaris iSCSI initiator.
Document 1175563.1 Sun Storage 7000 Unified Storage System: How to configure iSCSI CHAP for Windows iSCSI Initiator.

iSCSI MC/S (multiple connection per session) is not supported by ZFS Storage Appliance at this time, therefore only IPMP or Link Aggregation should be used for multiple network path at this time.

iSCSI Network Connectivity Information

The following are iSCSI related services and port numbers. The port number should be open to enable initiators, targets and iSCSI related services to communicate to each other.

  • iSCSI TCP/3260
  • iSNS TCP/3205
  • DNS TCP/53 UDP/53
  • RADIUS TCP/1812, TCP/1813

 

A note for iSER

If you are using Infiniband, ZFS Storage Appliance tries to use iSER (iSCSI Extention over RDMA) as default.  If your initiator does not support iSER or ZFSSA fails to establish iSER connection, it will use IPoIB (IP over Infiniband) instead and run iSCSI on the top of IPoIB.  The nodes without iSER support may perform slower than the nodes with iSER support.  ZFSSA shows iSCSI initiators with iSER support transparently in the iSCSI components.  Therefore, users can configure iSCSI without notifying they support iSER or not.

In order to identify if the node is running iSER or not, you can use the following mdb commands.  In case of ZFSSA, you need to go down to the Solaris shell, which is not allowed for the customers to use.

 -bash-4.1# echo "::iscsi_conn -v" |mdb -k

IDM Conn 10003105f000
           Conn Type: Initiator
           Transport: ISER_IB         <===
            Local IP: 192.168.050.022:0
           Remote IP: 192.168.050.100:3260
               State: 5
          Last State: 4
            Refcount: 2
      iSCSI Ini Conn: 10002e411000
      Parent Session: 100030852000
          Conn State: 3
     Last Conn State: 2
         Login Stage: 3
    Next Login Stage: 3
     Expected StatSN: 0x000000e3
   Active Queue Head: 0
    Abort Queue Head: 0

If you see Transport: SOCKETS instead of ISER_IB, you are running tcp (iSCSI) 

* To do list to myself --
Add the sample mdb output of below with how to read them
 - iSNS information
 - iSCSI Connectivity
 - iSCSI target
 - iSCSI service

 

Attention to  Linux Users

If your configuration matches the following, you may need a fix for  Bug 24516575 - target port group and rtpid change across heads cause ZFSSA connectivity issues.
  • Linux with multipath configuration using iSCSI
  • ZFS Storage is in a cluster configuration.
  • Takeover/Failback makes the path "stand-by."  (You can confirm this by running multipath -ll on Linux box.)

The fix is planned to be included in 8.7.5, however, an IDR is available.  Please contact to Oracle Support Team.

 

Click here to return to Document 1398581.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot iSCSI Problems.

References

<NOTE:1398633.1> - Sun Storage 7000 Unified Storage System: How to check iSNS/RADIUS settings on ZFSSA
<NOTE:1340245.1> - zpools From iSCSI Will Not Mount After Reboot
<NOTE:1489871.1> - Solaris Volume Manager (SVM) Mirrored Root Disk Server/System/Node Can Not Boot. Troubleshooting. Resolution Path
<NOTE:1539410.1> - Using 'init 5' or 'init 6' on a System with UFS or ZFS on iSCSI LUNs may hang and report "ISCSI_DOOR_DAEMON_SYSLOG_PP[nnnnn]: [ID nnnnnn daemon.emerg] @ iSCSI initiator service exited with sessions left."
<NOTE:1395320.1> - Sun Storage 7000 Unified Storage System: How to identify target IQN on ZFSSA
<NOTE:1395943.1> - Sun Storage 7000 Unified Storage System: How to identify the initiator group that IQN belongs to on ZFSSA
<NOTE:1395907.1> - Sun Storage 7000 Unified Storage System: How to identify the initiator IQN on ZFSSA
<NOTE:1401469.1> - Sun Storage 7000 Unified Storage System: How to identify target CHAP name on ZFSSA
<NOTE:1331769.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues
<NOTE:1395923.1> - Sun Storage 7000 Unified Storage System: How to check LUN properties on ZFSSA
<NOTE:1398581.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot iSCSI Problems
<NOTE:1175563.1> - Sun Storage 7000 Unified Storage System: How to configure iSCSI CHAP for Windows iSCSI Initiator
<NOTE:1401493.1> - Sun Storage 7000 Unified Storage System: How to identify initiator CHAP name on ZFSSA
<NOTE:1181788.1> - Sun Storage 7000 Unified Storage System: How to configure unidirectional CHAP for Solaris iSCSI initiator
<NOTE:1181843.1> - Sun Storage 7000 Unified Storage System: How to configure bidirectional CHAP for Solaris iSCSI initiator
<NOTE:1395356.1> - Sun Storage 7000 Unified Storage System: How to identify the target group that target IQN belongs on ZFSSA
<BUG:24516575> - TARGET PORT GROUP AND RTPID CHANGE ACROSS HEADS CAUSE ZFSSA CONNECTIVITY ISSUES

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback