Sun Storage 7000 Unified Storage System: How to troubleshoot Phone Home (i.e. scrk) connection issues

Asset ID:	1-71-1574071.1
Update Date:	2018-01-05
Keywords:

Solution Type Technical Instruction Sure

Solution 1574071.1 : Sun Storage 7000 Unified Storage System: How to troubleshoot Phone Home (i.e. scrk) connection issues

Applies to:

Sun ZFS Storage 7320 - Version All Versions and later
Sun Storage 7720 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)
How to troubleshoot phone home (i.e. scrk) connection issues

Goal

INTERNAL ONLY: The purpose of this document is to troubleshoot phone home, i.e. scrk connection issues.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

Solution

The first step is to check, using the appliance CLI, if phone home service has been configured.

>configuration services scrk show

If the phone home service is NOT activated, then you will receive the following output :

> configuration services scrk show
Properties:
                      <status> = disabled
                        soa_id =
                  soa_password =
                      proxy_on = false
                    proxy_host =
                    proxy_user =
                     domain_id =
                proxy_password =
                   domain_name =

To configure the Phone Home service, use the

CLI > configuration services scrk set <PARAMETER>

command to set the parameters required needed for the connection.

You'll need your SOA (Single-Sign On Account) identifier and password, also used to access SR on MOS.

Check with your network administration team, what proxy settings should be set to access the public network from any host of the local network.

If no proxy is needed for the local network, you will have to set parameters this way :

> configuration services scrk show
<status> = online
soa_id = aby.varghese@oracle.com
soa_password = *********
proxy_on = false
proxy_host =
proxy_user =
domain_id = 94275427
proxy_password =
domain_name = $aby.varghese@oracle.com

If a proxy is needed on the local network, then you will have to set parameters this way :

                      <status> = online
                        soa_id = arun.mathew@oracle.com
                  soa_password = *********
                      proxy_on = true
                    proxy_host = www-proxy-adc.us.oracle.com:80
                    proxy_user =
                     domain_id = 88457250
                proxy_password =
                   domain_name = $arun.mathew@oracle.com

Once the required parameters are set, if the appliance does not connect, here are the steps to troubleshoot the connection issue:

1. Try to set a proxy if not already done.

Phone Home service relies on the connection from the NAS box to the server asr-services.oracle.com on port 443.

443 is the regular HTTPS port, so if you find proxy settings from a host that can access internet on HTTPS protocol, then those settings should be work for Phone Home service.

2. From the appliance CLI, traceroute to the Oracle phone home server, and check we can go out of the local network.

>ping inv-cs.oracle.com

Oracle incorporates a firewall which blocks ICMP and both asr-services.oracle.com and inv-cs.oracle.com are subject to it. So ping will always show

> ping inv-cs.oracle.com
no answer from asr-services.oracle.com

Traceroute is expected to time out on hosts behind the firewall, but it is useful to verify that traceroute makes it out of customer network and into Oracle network as the ending will look similar to:

> traceroute inv-cs.oracle.com
traceroute to inv-cs.oracle.com (192.18.110.10), 30 hops max, 60 byte packets
...
8 * ae-21-52.car1.denver1.level3.net (4.69.147.99) 1.468 ms 1.426 ms
9 sun-microsy.car1.denver1.level3.net (4.53.0.122) 3.472 ms 3.436 ms 3.284 ms
10 129.150.0.82 (129.150.0.82) 3.444 ms 3.500 ms 3.418 ms
11 * * *
... <Oracle firewall is blocking at this point>

A typical case where we detect that we cannot go out of customer's local network :

zfs01sc1:> traceroute inv-cs.oracle.com
traceroute: Warning: Multiple interfaces found; using 10.7.81.78 @ igb0
traceroute to inv-cs.oracle.com (192.18.110.13), 30 hops max, 40 byte packets
1 * * *
zfs01sc1:>

In that case, the traceroute output above shows that the box cannot even reach its gateway.

The routing could also be checked in that case.

vzy7410a1:> configuration net routing show

Routes:

ROUTE      DESTINATION                      GATEWAY         INTERFACE TYPE
route-000 0.0.0.0/0                        10.166.58.1     nge0      static
route-003 10.166.58.0/23                   10.166.58.195   nge0      system

If none of the hints above help to diagnose the issue, contact the local network administrators and ask them to provide the right network settings to reach host asr-services.oracle.com on port 443.

3. It is a good test to ping transport.oracle.com.

ICMP requests to transport.oracle.com are not blocked.

This is expected to respond to ping due to firewall allowing ICMP to this one host.

> ping transport.oracle.com
transport.oracle.com is alive

traceroute will still show that we made it into an Oracle network :

> traceroute transport.oracle.com
traceroute: Warning: Multiple interfaces found; using 10.145.229.137 @ igb0
traceroute to transport.oracle.com (141.146.1.169), 30 hops max, 40 byte packets
1 x.x.x.x 0.393 ms 0.257 ms 0.225 ms
...

8 austin1q7-rtr-15-poc-4.us.oracle.com (144.21.45.186) 38.111 ms 40.216 ms 34.196 ms
9 austin1q7-rtr-24-te2-1.us.oracle.com (144.20.2.1) 33.521 ms 33.305 ms 33.354 ms
10 adcq7-cr-rtr-2-e-4-1.us.oracle.com (144.20.2.106) 33.615 ms 33.747 ms 33.677 ms
11 adcq12z19-fw6-i.us.oracle.com (10.231.231.28) 33.557 ms 33.592 ms 33.609 ms
12 lb-transport.oracle.com (141.146.1.169) 33.923 ms 33.929 ms 34.954 ms

For TSC Engineers, we can verify that we can connect to the ASR servers using ncat as we know all of these services operate on default https port of 443 from the shell in the following manner (and thus should be for Internal Only, as customer is not allowed to run shell commands):

# nc -vz asr-services.oracle.com 443
Connection to asr-services.oracle.com 443 port [tcp/*] succeeded!
# nc -vz inv-cs.oracle.com 443
Connection to inv-cs.oracle.com 443 port [tcp/*] succeeded!
# nc -vz transport.oracle.com 443
Connection to transport.oracle.com 443 port [tcp/*] succeeded!

The log report in /var/ak/logs/scrk.ak may contain some useful error messages in the case of connection failure (should be read with aklog ):

Here is the associated error message logged by the appliance :

#aklog /var/ak/logs/scrk.ak

                AKV_DOMAIN = sun.com
                AKV_GATE_ID = ak
                AKV_PRODUCT = SUNW,otoro
                AKV_COMPONENT = 2011.04.24.3.0
                AKV_BUILD = 1
                AKV_BRANCH = 1.19
                AKV_TIMESTAMP = 20120507T192643Z
                AKV_VERSION_BASE = 2011.04.24.3.0,1-1.19
                AKV_VERSION = 2011.04.24.3.0,1-1.19:20120507T192643Z
                AKV_FMRI = pkg://sun.com/ak/SUNW,otoro@2011.04.24.3.0,1-1.19:20120507T192643Z
                AKV_REQUIRED = 2010.08.17.2.1,1-1.21
                aksc_asn = 7a3f6d40-9f5a-ec46-e21a-ba0143537458
                aksc_csn = 1221FMJ00W
                aksc_ssn = 1221FMJ00W
                aksc_ak_product = SUNW,otoro
                aksc_version = ak/generic@2011.04.24.3.0,1-1.19
                aksc_isa = i386
                aksc_platform = i86pc
                aksc_hw_vendor = SUN MICROSYSTEMS
                aksc_hw_cpu_vendor = INTEL
                aksc_hostid = 0
                aksc_updated = 2012-05-26 23:15:44 GMT
                timestamp = 2013-06-20 00:17:58 UTC
                aksi_nodename = zfs01sc1
                activation_user = aby.varghese@oracle.com
                message_uuid = 9cc524d2-f944-4fda-bc97-f359ad0d009b
                message_time = 2013-06-20T00:17:58
                asset_id = urn:st:7a3f6d40-9f5a-ec46-e21a-ba0143537458
        (end arguments)

        result = (embedded nvlist)
        nvlist version: 0
                status = error_conn
                code = 0x0
                akerrno = 497
                akerrmsg = failed to make request: error connecting to support service (connect() timed out!)
        (end result)

This log file is also available in the support bundle in logs/scrk/ak.txt. But when Phone Home fails, most of the time, no bundle will be uploaded to cores, unless customer uploads it manually.

That could help determining if this network connection has ever worked, or if any change has occured on the system which may be resulted in failure.

Attachments

This solution has no attachment