Oracle ZFS Storage Appliance: BUI/CLI Inaccessible when 'Self-Replication' is Configured using a 127.0.0.1 IP Address

Asset ID:	1-72-2063635.1
Update Date:	2016-02-10
Keywords:

Solution Type Problem Resolution Sure

Solution 2063635.1 : Oracle ZFS Storage Appliance: BUI/CLI Inaccessible when 'Self-Replication' is Configured using a 127.0.0.1 IP Address

Applies to:

Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-BA - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

The BUI and CLI are not accessible after configuring replication between peer nodes in a cluster.

In gcore of akd, there is a lock shown in ::ak_rm_elem...

> ::ak_rm_lock
SOURCE

         address: 8897c50
             asn: 598bd819-6079-eeeb-abf6-a3743addf9b2
           flags: LOCKED
           owner: self
       local tid: 0xee
      remote tid: -

DEST

         address: 8897cd0
             asn: 598bd819-6079-eeeb-abf6-a3743addf9b2
           flags: 0
           owner: -
       local tid: -
      remote tid: -

DEST LOCKV[0]

         address: 8897d50
             asn: 598bd819-6079-eeeb-abf6-a3743addf9b2
           flags: LOCKED
           owner: self
       local tid: 0xee    <<<<<<<<<
      remote tid: -

DEST LOCKV[1]

         address: 8897dd0
             asn: c44bf7c6-8ec7-4097-f2ff-8850ffaad851
           flags: 0
           owner: -
       local tid: -
      remote tid: -

...This also can be seen in ::stacks...

> ::stacks

...

ee       PARKED   MX              1
         libc_hwcap1.so.1`mutex_lock_impl+0x166
         libc_hwcap1.so.1`mutex_lock+0x16
         libak.so.1`ak_peer_token_send_lock+0x33
         libak.so.1`ak_xdr_rpc_call+0x15b
         libak.so.1`com_sun_ak_rm_lockEnterId+0x3e
         libak.so.1`ak_rm_lock_enter+0x15c
         libak.so.1`ak_rm_enter_self+0x4b
         libak.so.1`ak_rm_iterate+0x92
         nas.so`com_sun_ak_nas_repl_listPools_svc+0x8c
         nas.so`com_sun_ak_nas_repl_listPools_svc_call+0x29
         libak.so.1`ak_xdr_svc_call+0xab
         libak.so.1`ak_xdr_rpc_svc+0x149
         libak.so.1`ak_peer_svc_xdr+0x67
         libak.so.1`ak_peer_rpc_svc+0x89
         libak.so.1`ak_rpc_svc_peer_process+0x38
         libak.so.1`ak_rpc_svc+0xea
         libak.so.1`ak_peer_conn_listener+0xfb
         libak.so.1`ak_thread_start+0x6a
         libc_hwcap1.so.1`_thrp_setup+0x9d
         libc_hwcap1.so.1`_lwp_start

The 'findstack' was...

> ee::findstack -v
stack pointer for thread ee: ee21f458
[ ee21f458 libc_hwcap1.so.1`__lwp_park+0x19() ]
ee21f498 libc_hwcap1.so.1`mutex_lock_impl+0x166(cf7e0a0, 0, fe337e04, fe337d5c, fe337d74, fe337d80)
ee21f4b8 libc_hwcap1.so.1`mutex_lock+0x16(cf7e0a0, ee21f570, ee21f4f8, f00fb759)
ee21f4f8 libak.so.1`ak_peer_token_send_lock+0x33(cf7e088, f019cabc, 36396463, f017da08)
ee21f5d8 libak.so.1`ak_xdr_rpc_call+0x15b(ee21f660, cf7e088, f01c1658, 1)
ee21f5f8 libak.so.1`com_sun_ak_rm_lockEnterId+0x3e(ee21f660, cf7e088, c6adcc42, eeeb00ee)
ee21f698 libak.so.1`ak_rm_lock_enter+0x15c(8136c08, 8897d50, ee21f6f0, f0104bd1)
ee21f6d8 libak.so.1`ak_rm_enter_self+0x4b(8136c08)
ee21f768 libak.so.1`ak_rm_iterate+0x92(8136c08, fd6ca938, fd6a80bc, fd678610, ee21f7b0, 0)
ee21f7d8 nas.so`com_sun_ak_nas_repl_listPools_svc+0x8c(cf7e088, ee21f810, 281381c8, fd69ecc3)
ee21f7f8 nas.so`com_sun_ak_nas_repl_listPools_svc_call+0x29(cf7e088, ee21f810, ee21fd20, feefc3a4)
ee21fd88 libak.so.1`ak_xdr_svc_call+0xab(8184ef0, 16c8e348, fd6cb7e8, ee21fdb0)
ee21fe18 libak.so.1`ak_xdr_rpc_svc+0x149(8184ef0, 16c8e348, 168c8d80, 40, ee21fe4c, 1)
ee21fe58 libak.so.1`ak_peer_svc_xdr+0x67(8136c08, 16c8e3e8, 16c8e348, 6, 168c8d80, 40)
ee21feb8 libak.so.1`ak_peer_rpc_svc+0x89(8136c08, 16c8e3e8, 6, 168c8d80, 40, 12032a08)
ee21fee8 libak.so.1`ak_rpc_svc_peer_process+0x38(8136c08, 16c8e3e8, 6, 168c8d80, 40, 12032a08)
ee21ff38 libak.so.1`ak_rpc_svc+0xea(8136c08, 16c8e3e8, 168c8c08, 12032a08, 0)
ee21ff98 libak.so.1`ak_peer_conn_listener+0xfb(16c8e348, 0, 0, f015cd3d)
ee21ffc8 libak.so.1`ak_thread_start+0x6a(16e0d480, fefa2000, ee21ffe8, fef03719)
ee21ffe8 libc_hwcap1.so.1`_thrp_setup+0x9d(f4c01140)
ee21fff8 libc_hwcap1.so.1`_lwp_start(f4c01140, 0, 0, 0, 0, 0)

Cause

This is bug 15683126 and is caused by having replication targets using an IP address of 127.0.0.1.

Solution

Remove replication targets that have IP address of 127.0.0.1.

Use a 'correctly-configured' interface/IP address for replication between the cluster nodes.

References

<BUG:21445016> - NAS_LISTPOOLS CAN HANG WHEN SETTING UP SELF-REPLICATION
<BUG:15683126> - SUNBT7002148 REVERSING SELF-REPLICATION PACKAGE CREATES DUPLICATE TARGETS

Attachments

This solution has no attachment