| Asset ID: |
1-72-1553935.1 |
| Update Date: | 2018-05-25 |
| Keywords: | |
Solution Type
Problem Resolution Sure
Solution
1553935.1
:
Sun Storage 7000 Unified Storage System: BUI/CLI hang when attempting to 'phone home' or upload supportbundle
| Related Items |
- Sun ZFS Storage 7320
- Sun Storage 7210 Unified Storage System
- Oracle ZFS Storage ZS3-BA
- Oracle ZFS Storage ZS5-4
- Oracle ZFS Storage ZS3-2
- Sun Storage 7410 Unified Storage System
- Oracle ZFS Storage ZS3-4
- Sun ZFS Storage 7420
- Oracle ZFS Storage ZS5-2
- Sun Storage 7310 Unified Storage System
- Oracle ZFS Storage ZS4-4
- Sun ZFS Storage 7120
- Sun Storage 7110 Unified Storage System
|
| Related Categories |
- PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
|
In this Document
Applies to:
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)
Symptoms
Unable to access BUI & CLI
root@sldcntapadm01 ~]# ssh auto@sl04stor43
Last login: Thu May 2 05:10:36 2013 from 10.217.123.45
Waiting for the appliance shell to start ...
The appliance shell is taking longer than usual to start.
Press Ctrl-C to exit.
Waiting for the appliance shell to start ...
aksh-wrapper: FATAL: Appliance shell failed to start in the allotted time
[root@sldcntapadm01 ~]#
System stopped phoning home with daily heartbeats
Possible DNS problem ?
adc26stor22# ping transport.oracle.com
transport.oracle.com is alive
adc26stor22# ping www-proxy-adc.us.oracle.com
www-proxy-adc.us.oracle.com is alive
adc26stor22# ping inv-cs.us.oracle.com
ping: unknown host inv-cs.us.oracle.com
adc26stor22# ping supportfiles.sun.com
no answer from supportfiles.sun.com
adc26stor22# ping asr-service.oracle.com
ping: unknown host asr-service.oracle.com
adc26stor22# nslookup inv-cs.us.oracle.com
Server:10.224.123.45
Address:10.224.123.45#66
** server can't find inv-cs.us.oracle.com: NXDOMAIN
Corefile analysis of AKD:
> ::stacks -c side_ ak_sidedoor
THREAD STATE SOBJ COUNT
c5 PARKED MX 4
libc_hwcap1.so.1`mutex_lock_impl+0x163
libc_hwcap1.so.1`mutex_lock+0x10
libak.so.1`akx_lock_common+0x53
libak.so.1`akx_lock+0x16
libak.so.1`akx_invoke+0x6e
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
c4 PARKED CV 3
libc_hwcap1.so.1`cond_wait_queue+0x60
libc_hwcap1.so.1`__cond_wait+0x86
libc_hwcap1.so.1`cond_wait+0x24
libc_hwcap1.so.1`pthread_cond_wait+0x21
workflow.so`ak_workflow_wait+0xb7
workflow.so`ak_workflow_result+0x22
workflow.so`ak_workflow_execute+0x5f9
libak.so.1`akx_invoke+0x9a
libak.so.1`ak_singlecall+0xb4f
libak.so.1`ak_multicall+0xe6
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
10c UNPARKED <NONE> 1
libc_hwcap1.so.1`poll+0x4c
libcurl.so.3.0.0`Curl_socket_ready+0xcd
libcurl.so.3.0.0`Transfer+0x2df
libcurl.so.3.0.0`Curl_do_perform+0xc1
libcurl.so.3.0.0`Curl_perform+0xae
libcurl.so.3.0.0`curl_easy_perform+0xa7
scrk.so`ak_scrk_curl_go+0xb1
scrk.so`ak_scrk_list_domains+0x1cc
scrk.so`ak_scrk_listDomains+0x10d
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_sidedoor+0x8e
libak.so.1`ak_door_serve+0x77
> ::stacks -c ak_frontdoor
THREAD STATE SOBJ COUNT
bf PARKED CV 4
libc_hwcap1.so.1`cond_wait_queue+0x60
libc_hwcap1.so.1`__cond_wait+0x86
libc_hwcap1.so.1`cond_wait+0x24
libc_hwcap1.so.1`pthread_cond_wait+0x21
workflow.so`ak_workflow_wait+0xb7
workflow.so`ak_workflow_execute+0x719
libak.so.1`akx_invoke+0x9a
libak.so.1`ak_singlecall+0xb4f
libak.so.1`ak_multicall+0xe6
libak.so.1`akx_invoke+0x9a
libak.so.1`akx_call+0x7a
libak.so.1`akx_rpc_svc+0xb2
libak.so.1`ak_rpc_svc_door_process+0x33
libak.so.1`ak_rpc_svc+0xea
libak.so.1`ak_frontdoor+0x6e
libak.so.1`ak_door_serve+0x77
libc_hwcap1.so.1`__door_return+0x4c
> 10c::findstack -v
stack pointer for thread 10c: e8c1e648
[ e8c1e648 libc_hwcap1.so.1`__pollsys+0x15() ]
e8c1e688 libc_hwcap1.so.1`poll+0x4c(e8c1e6c0, 1, 3e8, fedb12f2)
e8c1e6e8 libcurl.so.3.0.0`Curl_socket_ready+0xcd(190, ffffffff, 3e8, feda2b6e)
e8c1e748 libcurl.so.3.0.0`Transfer+0x2df(4151b988, e8c1e77f, 7, feda3866)
e8c1e798 libcurl.so.3.0.0`Curl_do_perform+0xc1(43d82008, 4, fed80b90, feda3b82)
e8c1e7d8 libcurl.so.3.0.0`Curl_perform+0xae(43d82008, ffffffff, e8c1e7f8, feda409e)
e8c1e7f8 libcurl.so.3.0.0`curl_easy_perform+0xa7(43d82008, 2711, e8a90a8, fcf85b1f)
e8c1e928 scrk.so`ak_scrk_curl_go+0xb1(8120c08, 43d82008, e8c1e95c, 86e1608)
e8c1ee98 scrk.so`ak_scrk_list_domains+0x1cc(e8c1ef90)
e8c1eee8 scrk.so`ak_scrk_listDomains+0x10d(e8c1ef90, 16fb7020, e8c1ef18, f01575dd)
e8c1ef28 libak.so.1`akx_invoke+0x9a(e8c1ef90, fcf90318, fcf9044c, 16fb7020, 0, 2)
e8c1ef78 libak.so.1`akx_call+0x7a(e8c1ef90, e8c1fc34, 1cb, f015ae69)
e8c1f8f8 libak.so.1`akx_rpc_svc+0xb2(8120c08, c8f5c6c, e8c1fc34, 1cb, 31903c08, 0)
e8c1f928 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, c8f5c6c, 0, e8c1fc34, 1cb, 31903c08)
e8c1f978 libak.so.1`ak_rpc_svc+0xea(8120c08, c8f5c6c, e8c1fabc, 31903c08, 0, 0)
e8c1f9f8 libak.so.1`ak_sidedoor+0x8e(c8f5c48, e8c1fabc, 343, 0, 0)
e8c1fa64 libak.so.1`ak_door_serve+0x77(c8f5c48, e8c1fabc, 343, 0, 0, f00a4f0c)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
> bf::findstack -v
stack pointer for thread bf: f061e3b8
[ f061e3b8 libc_hwcap1.so.1`__lwp_park+0x19() ]
f061e3e8 libc_hwcap1.so.1`cond_wait_queue+0x60(e21dd0c, fcf4a470, 0, feec7b46)
f061e428 libc_hwcap1.so.1`__cond_wait+0x86(e21dd0c, fcf4a470, f061e448, feec7c03)
f061e448 libc_hwcap1.so.1`cond_wait+0x24(e21dd0c, fcf4a470, 14, feec7c44)
f061e468 libc_hwcap1.so.1`pthread_cond_wait+0x21(e21dd0c)
f061e4a8 workflow.so`ak_workflow_wait+0xb7(e21dcd0, f061eb90, 340573c8)
f061e938 workflow.so`ak_workflow_execute+0x719(f061eb90, 14bae8e8, 6e69676f, f01575dd)
f061e978 libak.so.1`akx_invoke+0x9a(f061eb90, fcf4a1f8, fcf4a32c, 14bae8e8, 0, 0)
f061ea98 libak.so.1`ak_singlecall+0xb4f(f061eb90, 10e748e0, 0, 0, 0, 0)
f061eae8 libak.so.1`ak_multicall+0xe6(f061eb90, 16a2f220, f061eb18, f01575dd)
f061eb28 libak.so.1`akx_invoke+0x9a(f061eb90, f01c3320, f01c36c0, 16a2f220, 0, 2)
f061eb78 libak.so.1`akx_call+0x7a(f061eb90, f061f7f8, 606, f015ae69)
f061f4f8 libak.so.1`akx_rpc_svc+0xb2(8120c08, 8709c6c, f061f7f8, 606, 0, f061f688)
f061f528 libak.so.1`ak_rpc_svc_door_process+0x33(8120c08, 8709c6c, 0, f061f7f8, 606, 0)
f061f578 libak.so.1`ak_rpc_svc+0xea(8120c08, 8709c6c, f061f680, 0, f061f688, 0)
f061f5c8 libak.so.1`ak_frontdoor+0x6e(8709c48, f061f680, 77e, 0, 0)
f061f628 libak.so.1`ak_door_serve+0x77(8709c48, f061f680, 77e, 0, 0, f00a4f0c)
00000000 libc_hwcap1.so.1`__door_return+0x4c()
> fcf9044c::print akx_method_t
{
akxm_name = 0xfcf8c85c "listDomains"
akxm_func = scrk.so`ak_scrk_listDomains
akxm_help = 0xfcf8c830 "list possible SOA domains for registration"
akxm_signature = 0
akxm_flags = 0
}
> 16fb7020::akx_node -j
[ 'kandasamy.jayaraman@oracle.com', true, 'http', 'www-proxy-adc.us.oracle.com:80', '', '' ]
> $q
PLEASE NOTE: "Oracle Support is making changes to the way we receive customer data by establishing a new corporate data transport capability.
As a result, supportfiles.sun.com is set to retire in April 2014. To prepare for this change, we strongly encourage you to begin using one of the preferred methods for customer file uploads:
-
- My Oracle Support (files less than 2GB) via your web browser
- Diagnostic Assistant (Part of the STB bundle and RDA.)
- Standard tools (Filezilla, cURL, winSCP) based upload to transport.oracle.com
- Secure File Transfer (SFT) for proactive files
Please note that you should no longer use Supportfiles.sun.com, supportuploads.sun.com, sftp.oracle.com when uploading files to Oracle support.
-
- View the “Uploading support data to Service Requests” video.
- Visit Uploading Files to Oracle Support (DOC ID 1547088.2) for further information.
Cause
The problem seen during shared shell is that ping and DNS does not work to supportfiles.sun.com/inv-cs.us.oracle.com/asr-service.oracle.com
But we are able to ping the proxy www-proxy-adc.us.oracle.com
The issue is related to the following bugs:
Bug 15803914 - SUNBT7184231 sidedoor stuck in libcurl
Bug 15795655 - SUNBT7173781 FMA fault did not attempt to phone home
Bug 15820758 - SUNBT7173781 FMA fault did not attempt to phone home
We will need to restart AKD to make the BUI working again.
But still DNS issue needs to be resolved.
It seems one AKD thread is stuck dealing with curl which is responsible for sending heartbeats to ASR.
Solution
As a workaround, restart AKD to release the stuck thread.
From the CLI issue:
maintenance system restart
This will restart AKD and all current logins to the management CLI and BUI will be dropped.
For a permanent fix, upgrade the Appliance Firmware Release version to 2011.1.6.0 (or later).
Return to Document 1401282.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)
***Checked for relevance on 25-MAY-2018***
References
<BUG:15795655> - SUNBT7173781 FMA FAULT DID NOT ATTEMPT TO PHONE HOME
<BUG:15803914> - SUNBT7184231 SIDEDOOR STUCK IN LIBCURL
<NOTE:1401282.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)
Attachments
This solution has no attachment
