The status page (BUI or CLI) is unavailable - will not load, view or work. It freezes the UI. To continue using the UI requires reconnecting and the same problem continues.
Problems are that when connecting to the BUI and the status page loads, it hangs indefinitely - repeatedly.
Some threads are locked up in AKD (appliance management daemon) ... particularly involving dtrace/analytics.
There are some mutex locks on some of the threads.
> ::stacks -c ak_sidedoor
THREAD STATE SOBJ COUNT
d5 UNPARKED 5
libak.so.1`ak_rpc_svc_door_reply+0x9f
libak.so.1`akx_rpc_svc+0x201
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
ce PARKED MX 3
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`akx_lock_common+0x53
libak.so.1`akx_lock+0x16
libak.so.1`akx_invoke+0x6e
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
d1 PARKED MX 1
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`akx_lock_common+0x53
libak.so.1`akx_lock+0x16
libak.so.1`akx_invoke+0x6e
libak.so.1`ak_singlecall+0xb4f
libak.so.1`ak_multicall+0xe6
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
> d1::findstack -v
stack pointer for thread d1: e9a1e9e8
[ e9a1e9e8 libc_hwcap1.so.1`__lwp_park+0x19() ]
e9a1ea28 libc_hwcap1.so.1`mutex_lock_impl+0x163(f01c955c, 0, e9a1ea58, f011c272, 235a332c, fe660000)
e9a1ea48 libc_hwcap1.so.1`mutex_lock+0x10(f01c955c, f017c4f4, 67616964, f0160a91)
e9a1ea68 libak.so.1`akx_lock_common+0x53(f01c94f8, 0, e9a1eac8, f015ff60, f017c4f4, e9a1ea98)
e9a1ea88 libak.so.1`akx_lock+0x16(f01c94f8, 746573, 677661, f01600a1)
e9a1eac8 libak.so.1`akx_invoke+0x6e(e9a1ece0, f01c94f8, f01c9620, 52e62c08, 0, 0)
e9a1ebe8 libak.so.1`ak_singlecall+0xb4f(e9a1ece0, bf6a3f8, 0, 0, 0, 0)
e9a1ec38 libak.so.1`ak_multicall+0xe6(e9a1ece0, 4a57ff70, e9a1ec68, f01600a1)
e9a1ec78 libak.so.1`akx_invoke+0x9a(e9a1ece0, f01cd788, f01cdb30, 4a57ff70, 0, 2)
e9a1ecc8 libak.so.1`akx_call+0x7a(e9a1ece0, e9a1f980, 47e, f016395d)
e9a1f648 libak.so.1`akx_rpc_svc+0xb2(8120c08, 97f94ec, e9a1f980, 47e, 1807ae08, 0)
e9a1f678 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 97f94ec, 0, e9a1f980, 47e, 1807ae08)
e9a1f6c8 libak.so.1`ak_rpc_svc+0xea(8120c08, 97f94ec, e9a1f808, 1807ae08, 0, 0)
e9a1f748 libak.so.1`ak_sidedoor+0x8e(97f94c8, e9a1f808, 5f6, 0, 0)
e9a1f7b0 libak.so.1`ak_door_serve+0x77(97f94c8, e9a1f808, 5f6, 0, 0, f00a5aa8)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
> ce::findstack -v
stack pointer for thread ce: ea61efa8
[ ea61efa8 libc_hwcap1.so.1`__lwp_park+0x19() ]
ea61efe8 libc_hwcap1.so.1`mutex_lock_impl+0x163(f01c955c, 0, ea61f028, fee440b6, ea61f048, f017cb88)
ea61f008 libc_hwcap1.so.1`mutex_lock+0x10(f01c955c, fe999000, ea61f058, f0160a91)
ea61f028 libak.so.1`akx_lock_common+0x53(f01c94f8, 0, 0, 3a10be08, 61746164, 746573)
ea61f048 libak.so.1`akx_lock+0x16(f01c94f8, 706f7250, 69747265, f01600a1)
ea61f088 libak.so.1`akx_invoke+0x6e(ea61f0f0, f01c94f8, f01c9670, 0, 0, 2)
ea61f0d8 libak.so.1`akx_call+0x7a(ea61f0f0, ea61fd88, 76, f016395d)
ea61fa58 libak.so.1`akx_rpc_svc+0xb2(8120c08, 97f94ec, ea61fd88, 76, 21038808, 0)
ea61fa88 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 97f94ec, 0, ea61fd88, 76, 21038808)
ea61fad8 libak.so.1`ak_rpc_svc+0xea(8120c08, 97f94ec, ea61fc10, 21038808, 0, 0)
ea61fb58 libak.so.1`ak_sidedoor+0x8e(97f94c8, ea61fc10, 1ee, 0, 0)
ea61fbb8 libak.so.1`ak_door_serve+0x77(97f94c8, ea61fc10, 1ee, 0, 0, f00a5aa8)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
========================================================
> ::stacks -c ak_frontdoor
THREAD STATE SOBJ COUNT
c9 UNPARKED 1
libc_hwcap1.so.1`_filbuf+0xd3
libc_hwcap1.so.1`fgets+0xf8
libak.so.1`ak_log_syslog_iter+0x3e
libak.so.1`ak_log_check+0x13a
libak.so.1`ak_log_size+0x70
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_frontdoor+0x6e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
cb PARKED MX 1
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`ak_log_open+0x62
libak.so.1`ak_log_indexLength+0x4d
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_frontdoor+0x6e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
cd PARKED MX 1
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`akx_lock_common+0x53
libak.so.1`akx_lock+0x16
libak.so.1`akx_invoke+0x6e
libak.so.1`ak_singlecall+0xb4f
libak.so.1`ak_multicall+0xe6
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_frontdoor+0x6e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
ca PARKED MX 1
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`akx_lock_common+0x53
libak.so.1`akx_lock+0x16
libak.so.1`akx_invoke+0x6e
libak.so.1`ak_singlecall+0xb4f
libak.so.1`ak_singlecall+0x792
libak.so.1`ak_multicall+0xe6
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_frontdoor+0x6e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
> ca::findstack -v
stack pointer for thread ca: eb21a708
[ eb21a708 libc_hwcap1.so.1`__lwp_park+0x19() ]
eb21a748 libc_hwcap1.so.1`mutex_lock_impl+0x163(f01c955c, 0, eb21a778, f011c272, 0, 0)
eb21a768 libc_hwcap1.so.1`mutex_lock+0x10(f01c955c, f017c8a4, 0, f0160a91)
eb21a788 libak.so.1`akx_lock_common+0x53(f01c94f8, 0, eb21a7e8, f015ff60, f017c8a4, eb21a7b8)
eb21a7a8 libak.so.1`akx_lock+0x16(f01c94f8, 746573, 6e65706f, f01600a1)
eb21a7e8 libak.so.1`akx_invoke+0x6e(eb21ab20, f01c94f8, f01c9788, 24111d38, 0, 0)
eb21a908 libak.so.1`ak_singlecall+0xb4f(eb21ab20, 4fb6e088, 1, 0, 0, eb21aa00)
eb21aa28 libak.so.1`ak_singlecall+0x792(eb21ab20, 35b4cba8, 0, 0, 0, 0)
eb21aa78 libak.so.1`ak_multicall+0xe6(eb21ab20, 3a204118, eb21aaa8, f01600a1)
eb21aab8 libak.so.1`akx_invoke+0x9a(eb21ab20, f01cd788, f01cdb30, 3a204118, 0, 2)
eb21ab08 libak.so.1`akx_call+0x7a(eb21ab20, eb21b794, 466c, f016395d)
eb21b488 libak.so.1`akx_rpc_svc+0xb2(8120c08, 89d0cec, eb21b794, 466c, 1add0408, eb21b624)
eb21b4b8 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 89d0cec, 0, eb21b794, 466c, 1add0408)
eb21b508 libak.so.1`ak_rpc_svc+0xea(8120c08, 89d0cec, eb21b61c, 1add0408, eb21b624, 0)
eb21b558 libak.so.1`ak_frontdoor+0x6e(89d0cc8, eb21b61c, 47e4, 0, 0)
eb21b5c4 libak.so.1`ak_door_serve+0x77(89d0cc8, eb21b61c, 47e4, 0, 0, f00a5aa8)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
> cb::findstack -v
stack pointer for thread cb: eae1eb58
[ eae1eb58 libc_hwcap1.so.1`__lwp_park+0x19() ]
eae1eb98 libc_hwcap1.so.1`mutex_lock_impl+0x163(1b5e36e0, 0, 733c3e65, 0, 8121738, f01c5000)
eae1ebb8 libc_hwcap1.so.1`mutex_lock+0x10(1b5e36e0, 1b51ae48, 7274732f, f00c3608)
eae1f008 libak.so.1`ak_log_open+0x62(8120c08, 1b51ae48, 0, f00c51f1)
eae1f048 libak.so.1`ak_log_indexLength+0x4d(eae1f0f0, 44431048, 6e654c78, f01600a1)
eae1f088 libak.so.1`akx_invoke+0x9a(eae1f0f0, f01cad98, f01caf38, 44431048, 0, 2)
eae1f0d8 libak.so.1`akx_call+0x7a(eae1f0f0, eae1fd64, 9c, f016395d)
eae1fa58 libak.so.1`akx_rpc_svc+0xb2(8120c08, 89d0cec, eae1fd64, 9c, 1add0408, eae1fbf4)
eae1fa88 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 89d0cec, 0, eae1fd64, 9c, 1add0408)
eae1fad8 libak.so.1`ak_rpc_svc+0xea(8120c08, 89d0cec, eae1fbec, 1add0408, eae1fbf4, 0)
eae1fb28 libak.so.1`ak_frontdoor+0x6e(89d0cc8, eae1fbec, 214, 0, 0)
eae1fb94 libak.so.1`ak_door_serve+0x77(89d0cc8, eae1fbec, 214, 0, 0, f00a5aa8)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
> cd::findstack -v
stack pointer for thread cd: eaa1ec28
[ eaa1ec28 libc_hwcap1.so.1`__lwp_park+0x19() ]
eaa1ec68 libc_hwcap1.so.1`mutex_lock_impl+0x163(f01cc9ec, 0, eaa1ec98, f011c272, eaa1ecca, f0173ab4)
eaa1ec88 libc_hwcap1.so.1`mutex_lock+0x10(f01cc9ec, f0199574, 0, f0160a91)
eaa1eca8 libak.so.1`akx_lock_common+0x53(f01cc988, 0, eaa1ed08, f015ff60, f0199574, eaa1ecd5)
eaa1ecc8 libak.so.1`akx_lock+0x16(f01cc988, 61657200, 64, f01600a1)
eaa1ed08 libak.so.1`akx_invoke+0x6e(eaa1ef20, f01cc988, f01ccba0, 1a979670, 0, 0)
eaa1ee28 libak.so.1`ak_singlecall+0xb4f(eaa1ef20, 2d7ca948, 0, 0, 0, 0)
eaa1ee78 libak.so.1`ak_multicall+0xe6(eaa1ef20, 2e182ac8, eaa1eea8, f01600a1)
eaa1eeb8 libak.so.1`akx_invoke+0x9a(eaa1ef20, f01cd788, f01cdb30, 2e182ac8, 0, 2)
eaa1ef08 libak.so.1`akx_call+0x7a(eaa1ef20, eaa1fb88, 276, f016395d)
eaa1f888 libak.so.1`akx_rpc_svc+0xb2(8120c08, 89d0cec, eaa1fb88, 276, 1add0408, eaa1fa18)
eaa1f8b8 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 89d0cec, 0, eaa1fb88, 276, 1add0408)
eaa1f908 libak.so.1`ak_rpc_svc+0xea(8120c08, 89d0cec, eaa1fa10, 1add0408, eaa1fa18, 0)
eaa1f958 libak.so.1`ak_frontdoor+0x6e(89d0cc8, eaa1fa10, 3ee, 0, 0)
eaa1f9b8 libak.so.1`ak_door_serve+0x77(89d0cc8, eaa1fa10, 3ee, 0, 0, f00a5aa8)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
========================================================
> f01c955c::mutex_owner
c5
> c5::findstack -v
stack pointer for thread c5: ebe753d8
[ ebe753d8 libc_hwcap1.so.1`__lwp_park+0x19() ]
ebe75418 libc_hwcap1.so.1`mutex_lock_impl+0x163(f01cc9ec, 0, ebe75498, feec4786, 5, f4e8f340)
ebe75438 libc_hwcap1.so.1`mutex_lock+0x10(f01cc9ec, fef61000, ebe75468, f0160a91)
ebe75458 libak.so.1`akx_lock_common+0x53(f01cc988, 0, ebe75488, feec3c16, fcaba100, f01c5000)
ebe75478 libak.so.1`akx_lock+0x16(f01cc988, fef61000, ebe754c8, f01610c3)
ebe75498 libak.so.1`akx_dependency_lock+0x27(f01c94f8, 1010000, 10000, f009613c)
ebe75558 libak.so.1`ak_dataset_tick+0x52(ebe75570, ebe75f10, f01c94f8, f0162b10)
ebe75ed8 libak.so.1`akx_timer+0x79(8120c08, ebe75f10, f01c94f8)
ebe75f48 libak.so.1`ak_timer_callout_execute+0xe6(8120c08)
ebe75f98 libak.so.1`ak_timer+0x63(9c28c28, 0, 0, f013930d)
ebe75fc8 libak.so.1`ak_thread_start+0x7b(a985640, fef61000, ebe75fe8, feecdcee)
ebe75fe8 libc_hwcap1.so.1`_thrp_setup+0x9b(f4e7d340)
ebe75ff8 libc_hwcap1.so.1`_lwp_start(f4e7d340, 0, 0, 0, 0, 0)
> f01cc9ec::mutex_owner
c6
========================================================
They all seems to be waiting for this thread.
> c6::findstack -v
stack pointer for thread c6: ebd763d8
[ ebd763d8 libc_hwcap1.so.1`ioctl+0x15() ]
ebd76438 libdtrace.so.1`dtrace_consume+0x1d0(18783008, 0, f01074bc, 0, 9fa8648, 9fa8648)
ebd76488 libak.so.1`ak_stat_dprocess+0xa0(8120c08, 9fa8648, 0, f0106d0d)
ebd764d8 libak.so.1`ak_stat_process+0x173(8120c08, 9fa8648, ebd76f10, f0106b1e)
ebd76518 libak.so.1`ak_statmod_process+0x9e(a1676c8, ebd76570, ebd76f10, f0108633)
ebd76558 libak.so.1`ak_stat_tick+0x276(ebd76570, ebd76f10, f01cc988, f0162b10)
ebd76ed8 libak.so.1`akx_timer+0x79(8120c08, ebd76f10, f01cc988)
ebd76f48 libak.so.1`ak_timer_callout_execute+0xe6(8120c08)
ebd76f98 libak.so.1`ak_timer+0x63(a35e8e8, 0, 0, f013930d)
ebd76fc8 libak.so.1`ak_thread_start+0x7b(a985870, fef61000, ebd76fe8, feecdcee)
ebd76fe8 libc_hwcap1.so.1`_thrp_setup+0x9b(f4e7f340)
ebd76ff8 libc_hwcap1.so.1`_lwp_start(f4e7f340, 0, 0, 0, 0, 0)
Once the management interface comes back online, go into the analytics datasets and reduce the amount of datasets in the list.
<NOTE:1401282.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)