Oracle ZFS Storage Appliance: Private Interface in tpg-tags May Cause iSCSI Service to Fail

Asset ID:	1-72-2164548.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2164548.1 : Oracle ZFS Storage Appliance: Private Interface in tpg-tags May Cause iSCSI Service to Fail

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS3-BA - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS4-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

ZFS Storage Appliance iSCSI service occasionally stops responding.

This may be reflected on the iSCSI client by system log entries such as:

Jun 6 00:18:40 linux1a kernel: connection20:0: detected conn error (1020)
Jun 6 00:18:41 linux1a iscsid: Target requests logout within 20 seconds for connection
Jun 6 00:18:44 linux1a iscsid: connect to 10.10.102.123:3260 failed (Connection refused)

It can be confirmed that the iSCSI service on the appliance is no longer accepting connections:

telnet <appliance-iscsi-target-IP> 3260
Trying xx.yy.zz.abc...
telnet: connect to address xx.yy.zz.abc: Connection refused
telnet: Unable to connect to remote host

In netstat -an, there will be no entry like the one below showing a listener on port 3260:

*.3260 *.* 0 0 262300 0 LISTEN

Matching the timestamp reported on the iscsi client, there should be a refresh of the private network datalink and accompanying network services.

For example, under svcs/logs:

network-routing-route:default.log: [ May 26 07:47:17 Executing stop method (:kill). ]
network-routing-route:default.log: [ May 26 07:47:17 Stopping because service disabled. ]
appliance-kit-network-interface:ixgbe3.log: [ May 26 07:47:20 Executing stop method ("exec /usr/lib/ak/svc/method/akinterface stop"). ]
appliance-kit-network-interface:ixgbe3.log:     [ May 26 07:47:20 Method "stop" exited with status 0. ]
appliance-kit-network-interface:ixgbe3.log:     [ May 26 07:47:20 Stopping because service disabled. ]
appliance-kit-network-datalink:ixgbe3.log:    [ May 26 07:47:21 Executing refresh method ("exec /usr/lib/ak/svc/method/akdatalink refresh"). ]
appliance-kit-network-datalink:ixgbe3.log:    [ May 26 07:47:21 Rereading configuration. ]
appliance-kit-network-interface:ixgbe3.log: [ May 26 07:47:23 Enabled. ]
appliance-kit-network-interface:ixgbe3.log: [ May 26 07:47:23 Executing start method ("exec /usr/lib/ak/svc/method/akinterface start"). ]
appliance-kit-network-datalink:ixgbe3.log:    [ May 26 07:47:23 Method "refresh" exited with status 0. ]
network-routing-route:default.log: [ May 26 07:47:24 Enabled. ]
network-routing-route:default.log: [ May 26 07:47:24 Executing start method ("exec /lib/svc/method/svc-route"). ]
appliance-kit-network-interface:ixgbe3.log:     [ May 26 07:47:24 Method "start" exited with status 0. ]
network-routing-route:default.log: [ May 26 07:47:25 Method "start" exited with status 0. ]

Cause

A possible cause for this is that a private network interface has mistakenly been included in the target portal groups for an iSCSI target.

A refresh of this private interface then causes the iSCSI service to hang.

The inclusion of the private interface in the target portal groups can be verified:

In the BUI

Under Configuration -> SAN -> iSCSI select the Edit icon for each iSCSI target.
In the pop-up, under Network Interfaces, look for the private interface.

In the CLI

Under "configuration san iscsi targets", for each target run "select target-00x show".
Look for the private interface under "interfaces".

In the support bundle, this can be verified in the file iscsi/iscsi_targets.out under the tpg-tags property for each target.

If tpg-tags is set to "default" it includes all portals listed in iscsi/iscsi_tpgs.out.

The above outputs can be obtained with

CLI> confirm shell itadm list-target -v
CLI> confirm shell itadm list-tpg -v

Solution

To immediately recover iSCSI functionality, restart the iSCSI service

In the BUI, under Configuration Services, click on the Restart Service icon for the iSCSI service.

In the CLI, run

> configuration services iscsi disable
> configuration services iscsi enable

To avoid the issue in the future, remove private network interfaces from the target portal groups of all iSCSI targets.

In the BUI

Under Configuration -> SAN -> iSCSI select the Edit icon for each iSCSI target.
In the pop-up, under Network Interfaces, select only the interfaces that should be used for iSCSI traffic. Use Shift-select for several choices.

In the CLI

Under "configuration san iscsi targets", select each target and define the list of interfaces correctly using:

configuration san iscsi targets target-000> set interfaces=<interface_list>
interfaces = xxxx0 yyyy1 (uncommitted)
configuration san iscsi targets target-000> commit

With the implementation of he RFE in bug 20350607 this situation should no longer be possible.

References

<BUG:20350607> - ISCSI TARGET CONFIG SHOULD REMAIN IN SYNC WITH NETWORK CONFIG CHANGES
<BUG:23708005> - PORT 3260 STOPS LISTENING AFTER REFRESH OF PRIVATE INTERFACE IN TPG

Attachments

This solution has no attachment