Snapper - ILOM Snapshot Summary Tool

Asset ID:	1-79-1359411.1
Update Date:	2018-03-14
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1359411.1 : Snapper - ILOM Snapshot Summary Tool

Applies to:

SPARC T3-1 - Version Not Applicable and later
Sun SPARC Enterprise T5120 Server - Version Not Applicable and later
Sun Netra T5220 Server - Version Not Applicable and later
Information in this document applies to any platform.

Purpose

This lists how to run snapper on an ILOM snapshot & provides sample output.

Scope

Details

ILOMs exist on T5xx0 series & newer SPARC servers (& on many X64 servers). ILOM snapshots contain a great deal of SP failure information. Snapper gathers hardware related information from an ILOM snapshot & places the output in files HWsummary.txt & HWsummary.html which are typically placed into the snapshot's top directory unless redirected as shown below. Most output is filtered to only show erroneous entries or short status of various components. This tool consists of: "snap.sh" which performs some UNIX commands & then calls buf2csth-sparc + a compiled C program "snap". snap then parses the snapshot for hardware information. I have the latest SPARC & X64 Solaris version of this tool on ISDE, & typically a known working (older version) loaded on beehive.

On an ISDE terminal from the top directory of the snapshot, typing "snapper ." executes the program. Some examples are as follows:

   snapper snapshot-top-directory
   snapper . (if already in the top directory of the snapshot)
   snapper . output-directory

FEs can access the beehive URL listed above to download the C executable: snapx + the script: snapx.sh + buf2csth-x86 to one directory on a Solaris based laptop. If this directory is added to the PATH variable, it is run by typing "snapx.sh ." while in the snapshot's top level directory.

Snapper can also be executed via the Collection Viewer's "Available Analyzer" link when positioned in the Snapshot's top level directory.

*****************************************************************************************
snapper revision 4.47 (Snapshot Revision: 1.1) 16/01/14-00:19:12
SR #: 3-12010130931 Platform: SPARC T7-1 Serial#: AK00341234
*****************************************************************************************

Internal: System Config Fault Info Logs Analysis
External: Snapper Issues FW Troubleshooting SSH

========== System Configuration ============

Lists platform FW versions & if NTP in use
##### ilom/@usr@local@bin@collect_properties.out #####

ILOM FW: 3.2.5.8.g r105871
System FW: 9.5.2.g 2015/12/07 10:10
ILOM NTP: disabled

##### ilom/@usr@local@bin@hw_version_-local.out #####

0. /SYS/MB/FPGA (FPGA) FPGA Version: 10.2.1.8
1. /SYS/MB/CM/FPGA (CPUMFPGA) FPGA Version: 3.2.3.5
2. /SYS/MB/CPU (CPU) (CPU 0) SN: 0000000000000000000bb70882904181 cores: 32
3. /SYS/MB/CM/VCORE_OBPS_0 (D253) Revision: 0x03:0x77
4. /SYS/MB/CM/VCORE_OBPS_1 (D253) Revision: 0x03:0x77
34. /SYS/MB/BOB_VCORE_OBPS0 (UDT020A0X3) Version: UDT020A_0 (778b9c6705e9973a)
40. /SYS/MB/BOB_VCORE_OBPS1 (UDT020A0X3) Version: UDT020A_0 (778b9c6705e9973a)
46. /SYS/MB/CM/CMP/MR0/BOB_VCORE_OBPS (UDT020A0X3) Version: UDT020A_0 (778b9c6705e9973a)
52. /SYS/MB/CM/CMP/MR1/BOB_VCORE_OBPS (UDT020A0X3) Version: UDT020A_0 (778b9c6705e9973a)

Lists Process information & Memory/Swap sizes.
##### spos_info/@proc@uptime #####

Up Time: 525700.51 Seconds 6.08 Days
Idle Time: 458904.44 Seconds 5.31 Days
ILOM CPU usage: 13%.

##### spos_info/@usr@bin@top_-bn_2.out #####

Mem: 144888K used, 343440K free, 0K shrd, 13116K buff, 46528K cached
Load average: 0.64 0.71 0.61 2/246 28071
PID PPID USER STAT VSZ %MEM %CPU COMMAND
715 1 root S 99596 20% 6% /usr/local/bin/capidirectd
1039 1 root S 163m 34% 0% /usr/local/bin/health
1190 1 root S 146m 31% 0% /usr/local/bin/pod
1148 1 root S 136m 29% 0% /usr/local/bin/hostd
1179 1 root S 86352 18% 0% /usr/local/bin/pdm
1222 1 root S 67884 14% 0% /usr/local/bin/fetd
1346 1 root S 58684 12% 0% /usr/local/bin/mediator -f /etc/mediat
1732 1637 root S 56644 12% 0% /usr/local/bin/snmpd -A -f -Le -C -c/e
1122 1098 root S 47688 10% 0% [MsgHndlr]

##### spos_info/@proc@meminfo #####
MemTotal: 488328 kB
MemFree: 343332 kB

Check size of /persist & /coredump entries only.
##### spos_info/@bin@df_-k.out If high /coredump size upgrade passed ILOM: 3.1.2.20.c (bug 17265880) #####

Filesystem 1K-blocks Used Available Use% Mounted on
ubi0:persist 95752 1812 88880 2% /persist
ubi0:coredump 61148 1188 56648 2% /coredump

##### fma/errlog.txt (located in /persist) #####

-rwxrwxrwx+ 1 iamd staff 0 Jan 14 12:01 errlog.txt

##### ilom/conf/@conf@interfaces #####

##### spos_info/net/@sbin@ifconfig_-a.out #####

EthUsb0 UP 169.001.002.003 RX Pkts: 349807 errors: 0 TX Pkts: 350095 errors: 0
eth0 UP 10.001.002.003 RX Pkts: 242343 errors: 1 TX Pkts: 156115 errors: 7

##### ilom/@usr@local@bin@collect_properties.out See doc 1610270.1 #####

/SP/powermgmt policy = performance

Listing of board part #s & serial #s
##### ilom/@usr@local@bin@collect_properties.out #####
-------------- FRU ------------- - Part No - ----- PPart No ----- ----- Serial # ----- --- Mfg ---- ------ Product ----- Status
DBP 7097205-04 489089M+15356L1LU8 Oracle Corpo OK
HDD0 H101860SFSUN600G 001545F9AJZC HGST
HDD1 H101860SFSUN600G 001545F9B20C HGST
MB 7315713-01 465769T+1549N201E8 Oracle Corpo OK
MB/CM 7315713-01 465769T+1549N201E8 Oracle Corpo
P/B01/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFBA7 Samsung 32768MB DDR4 SDRAM D OK
P/B01/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFB25 Samsung 32768MB DDR4 SDRAM D OK
P/B11/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFB61 Samsung 32768MB DDR4 SDRAM D OK
P/B11/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE26 Samsung 32768MB DDR4 SDRAM D OK
P/B21/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE7B Samsung 32768MB DDR4 SDRAM D OK
P/B21/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE3E Samsung 32768MB DDR4 SDRAM D OK
P/B31/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFAE4 Samsung 32768MB DDR4 SDRAM D OK
P/B31/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE17 Samsung 32768MB DDR4 SDRAM D OK
P0/M0 7300948-02 465769T+1509N50149 Oracle Corpo OK
P/M0/B20/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFB30 Samsung 32768MB DDR4 SDRAM D OK
P/M0/B20/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE61 Samsung 32768MB DDR4 SDRAM D OK
P/M0/B30/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFB46 Samsung 32768MB DDR4 SDRAM D OK
P/M0/B30/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFDBC Samsung 32768MB DDR4 SDRAM D OK
P0/M1 7300948-02 465769T+1509N5011A Oracle Corpo OK
P/M1/B00/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE2C Samsung 32768MB DDR4 SDRAM D OK
P/M1/B00/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE89 Samsung 32768MB DDR4 SDRAM D OK
P/M1/B10/C0/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFE45 Samsung 32768MB DDR4 SDRAM D OK
P/M1/B10/C1/D0 7082851-01 M386A4G40DM0-CPB 00CE011450026DFC74 Samsung 32768MB DDR4 SDRAM D OK
SP 7319380-01 465769T+1544NM0883 Oracle Corpo OK
PS0 7044130-99 465824T+1508C35536 6580 DELTA E PS OK
PS1 7044130-99 465824T+1508C35531 6580 DELTA E PS OK
PCIE1 7104074
PCIE6 X1109A-Z

Listing of failed sensors (if any)
##### ipmi/@usr@local@bin@ipmiint_sensor_list.out #####

##### ipmi/@usr@local@bin@ipmiint_sdr_list_all.out (removes ok entries) #####

========== Fault Information ============

Lists if any ILOM core files exist
##### ILOM Dump files: ilom/traces/ #####

@coredump@sp_trace@reboot@dump.gz Wed Jan 13 20:13:35 2016
@coredump@sp_trace@reboot@dump.0.gz Wed Jan 13 20:13:35 2016

##### fma/@persist@faultdiags@ereports.log Skips benign & repeated ereports. #####

2016-01-05/13:37:51 ereport.chassis.tli.ok@/SYS
2016-01-05/13:38:16 ereport.sp.boot-cold@/SYS/MB/SPM
2016-01-05/13:38:37 ereport.chassis.sp.restart@/SYS/MB/SPM
2016-01-05/13:38:37 ereport.chassis.sp.post.ethernet.linkstatus@/SYS/MB/SPM/PHY0_LINKSTATUS
2016-01-05/13:39:04 ereport.chassis.device.psu.ext-ac-fail@/SYS/PS0 /SYS/PS0/STATE
2016-01-05/13:39:09 ereport.chassis.config.psu.ok@/SYS/PS1 /SYS/PS1/STATE
2016-01-06/14:48:58 ereport.chassis.tli.ok@/SYS
2016-01-06/14:49:24 ereport.sp.boot-cold@/SYS/MB/SPM
2016-01-06/14:49:47 ereport.chassis.sp.restart@/SYS/MB/SPM
2016-01-06/14:49:47 ereport.chassis.sp.post.ethernet.linkstatus@/SYS/MB/SPM/PHY0_LINKSTATUS
2016-01-06/14:50:14 ereport.chassis.config.psu.ok@/SYS/PS0 /SYS/PS0/STATE
2016-01-07/11:34:36 ereport.chassis.device.psu.ext-ac-fail@/SYS/PS0 /SYS/PS0/STATE
2016-01-07/11:51:47 ereport.chassis.tli.ok@/SYS
2016-01-07/11:52:12 ereport.sp.boot-cold@/SYS/MB/SPM
2016-01-07/11:52:33 ereport.chassis.sp.restart@/SYS/MB/SPM
2016-01-07/11:52:34 ereport.chassis.sp.post.ethernet.linkstatus@/SYS/MB/SPM/PHY0_LINKSTATUS
2016-01-07/11:53:06 ereport.chassis.config.psu.ok@/SYS/PS0 /SYS/PS0/STATE
2016-01-07/14:18:58 ereport.chassis.tli.ok@/SYS
2016-01-07/14:19:24 ereport.sp.boot-cold@/SYS/MB/SPM
2016-01-07/14:19:47 ereport.chassis.sp.restart@/SYS/MB/SPM
2016-01-07/14:19:47 ereport.chassis.sp.post.ethernet.linkstatus@/SYS/MB/SPM/PHY0_LINKSTATUS

##### elogs/@usr@local@bin@elogs_-eV.out #####

No entries in fmdump.

##### fma/@usr@local@bin@fmdump_-v.out Limit of 10 faults/day listed. #####

2016-01-05/18:00:31 b3d26df3-aaee-cc66-a5f4-9a6d5537e106 SPT-8000-5X
FRU = /SYS/PS1
2016-01-05/18:01:11 a2f82a33-384e-6e59-b8a0-9c0dc1c47bf2 ILOM-8000-81
FRU = /SYS/MB/SPM
2016-01-05/18:10:11 f08df7dd-5266-6eda-8d20-b5c333204502 ILOM-8000-9W
FRU = /SYS
ASRU = /SYS/MB/SPM
2016-01-05/19:03:10 05cf594f-b2d4-6102-e36c-9bcd1b157d72 ILOM-8000-81
FRU = /SYS/MB/SPM
2016-01-05/19:11:02 7850b047-9468-626e-fe0e-d38934767dfc SPT-8000-5X
FRU = /SYS/PS1
2016-01-05/13:39:14 c661c625-0a81-4acf-b898-aad155270782 SPT-8000-5X
FRU = /SYS/PS0
2016-01-07/11:34:46 24d4096a-15e7-c375-ef70-afc99a7e7ba0 SPT-8000-5X
FRU = /SYS/PS0

##### fma/@usr@local@bin@fmadm_faulty.out Prob Status 'solved' indicates FMA diagnosed the problem, but is not resolved!!! #####

2016-01-05/18:10:11 f08df7dd-5266-6eda-8d20-b5c333204502 ILOM-8000-9W Minor
MsgID: ILOM-8000-9W Minor The ILOM Mini-Root system is missing.

========== Logs ============

Lists BBR events of interest
##### ilom/bbr/bbr1.csv ##### This data is used only when FMA diagnosis is in question.
16/01/05-18:00:30 Header: 216 columns. /SP/CPU/* in column 75.
16/01/05-18:00:30 Sensors near 0 for 3 periods: /SYS/PS1/INPUT_POWER /SYS/PS1/I_+3V3 /SYS/PS1/V_+12V_STBY /SYS/PS1/V_IN
16/01/05-18:19:30 Header: 204 columns. /SP/CPU/* in column 75.
16/01/05-18:20:30 Header: 216 columns. /SP/CPU/* in column 75.
16/01/05-18:20:30 Sensors near 0 for 3 periods: /SYS/PS1/INPUT_POWER /SYS/PS1/I_+3V3 /SYS/PS1/V_+12V_STBY /SYS/PS1/V_IN
16/01/05-18:29:30 Header: 368 columns. /SP/CPU/* in column 227.
16/01/05-18:34:30 Header: 368 columns.
16/01/05-18:39:30 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW /SYS/PS1/INPUT_POWER /SYS/PS1/I_+3V3 /SYS/PS1/OUTPUT_POWER /SYS/PS1/V_+12V /SYS/PS1/V_+12V_STBY /SYS/PS1/V_IN
16/01/05-18:48:31 Header: 216 columns. /SP/CPU/* in column 75.
16/01/05-18:48:31 Sensors near 0 for 3 periods: /SYS/PS1/INPUT_POWER /SYS/PS1/I_+3V3 /SYS/PS1/V_+12V_STBY /SYS/PS1/V_IN
16/01/05-19:13:06 Header: 210 columns. /SP/CPU/* in column 75.
16/01/05-19:16:06 Header: 370 columns. /SP/CPU/* in column 229.
16/01/05-19:17:06 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW /SYS/PS1/INPUT_POWER /SYS/PS1/I_+3V3 /SYS/PS1/OUTPUT_POWER /SYS/PS1/V_+12V /SYS/PS1/V_+12V_STBY /SYS/PS1/V_IN
16/01/05-21:39:19 Header: 216 columns. /SP/CPU/* in column 75.
16/01/05-21:39:19 Possible reboot or hang. 1/2 hour+ time gap since 16/01/05-19:21:06.

16/01/06-22:50:26 Possible reboot or hang. 1/2 hour+ time gap since 16/01/05-21:40:19.

16/01/07-22:20:37 Possible reboot or hang. 1/2 hour+ time gap since 16/01/07-21:26:27.

16/01/09-00:35:09 Header: 222 columns. /SP/CPU/* in column 81.
16/01/09-00:36:09 Header: 368 columns. /SP/CPU/* in column 227.
16/01/09-00:40:09 Header: 368 columns.
16/01/09-00:44:09 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW
16/01/12-22:51:08 Last Date

##### ilom/bbr/bbr2.csv ##### This data is used only when FMA diagnosis is in question.
16/01/12-22:52:08 Header: 370 columns. /SP/CPU/* in column 229.
16/01/12-22:52:08 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW
16/01/12-23:36:09 Header: 369 columns. /SP/CPU/* in column 228.
16/01/12-23:38:09 Header: 370 columns. /SP/CPU/* in column 229.
16/01/12-23:38:09 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW
16/01/14-00:21:42 Last Date

##### ipmi/@usr@local@bin@ipmiint_sel_elist.out #####

1 | 01/05/2016 | 19:10:52 | Power Supply PS1/STATE | Power Supply AC lost | Asserted
2 | 01/05/2016 | 19:12:28 | System Boot Initiated | System Restart | Asserted
3 | 01/05/2016 | 21:39:04 | Power Supply PS0/STATE | Power Supply AC lost | Asserted
4 | 01/07/2016 | 19:34:36 | Power Supply PS0/STATE | Power Supply AC lost | Asserted
5 | 01/09/2016 | 00:33:51 | System Boot Initiated | System Restart | Asserted

##### ilom/@usr@local@bin@spshexec_show_-script_@X@logs@event@list.out Most session entries removed #####

104 Wed Jan 13 14:37:33 2016 System Log minor Host: Solaris running
100 Wed Jan 13 14:36:15 2016 System Log minor Host: Solaris rebooting
99 Wed Jan 13 14:35:49 2016 System Log minor Host: Solaris rebooting
90 Tue Jan 12 15:40:45 2016 System Log minor Host: Solaris booting
85 Tue Jan 12 15:39:07 2016 System Log minor Host: HV started
84 Tue Jan 12 15:34:19 2016 System Log minor Host: Powered On
83 Tue Jan 12 15:34:14 2016 System Log minor Host: Standby
82 Tue Jan 12 15:34:13 2016 Power Reset major /SYS has been reset by: Web session
81 Tue Jan 12 15:34:13 2016 Power Reset major /SYS has been reset by: Web session
80 Tue Jan 12 15:22:04 2016 System Log minor Host: Solaris running
73 Mon Jan 11 11:09:32 2016 System Log minor Host: Solaris booting
70 Fri Jan 8 17:10:00 2016 System Log minor Host: OpenBoot Running
67 Fri Jan 8 17:09:29 2016 System Log minor Host: HV started
66 Fri Jan 8 16:33:53 2016 System Log minor Host: Powered On
64 Fri Jan 8 16:33:51 2016 Power On major Power to /SYS has been turned on by: Web session, Username:root
63 Thu Jan 7 14:19:50 2016 System Log minor Host: Standby
62 Thu Jan 7 14:19:47 2016 Fault Warning minor NET MGMT port 0 cable is missing/broken or inactive.
61 Thu Jan 7 11:53:06 2016 Fault UUID_Repaired minor Fault with UUID 24d4096a-15e7-c375-ef70-afc99a7e7ba0 repaired
60 Thu Jan 7 11:53:06 2016 Fault Repair minor Component /SYS/PS0 repaired
59 Thu Jan 7 11:53:06 2016 Fault Repair minor Fault fault.chassis.env.power.loss on component /SYS/PS0 cleared
58 Thu Jan 7 11:52:37 2016 System Log minor Host: Standby
57 Thu Jan 7 11:52:34 2016 Fault Warning minor NET MGMT port 0 cable is missing/broken or inactive.
56 Thu Jan 7 11:34:46 2016 Fault Fault critical Fault detected at time = Thu Jan 7 11:34:46 2016. The suspect component: /SYS/PS0 has fault.chassis.env.power.loss with probability=100. Refer to http://support.oracle.com/msg/SPT-8000-5X for details.

##### ilom/statistics/@usr@local@bin@statistics_-p.out Check for CPU throttling. #####

##### spos_logs/@var@log@messages Benign & repeated messages filtered #####

Jan 7 14:19:05 ORACLESP-AK00340758 syslogd 1.5.0: restart.
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0x13c/0x224()
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: NETDEV WATCHDOG: EthUsb0 (): transmit timed out
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: Modules linked in: Timer video KhUsb dram_ecc sptrace(P) usb i2c_pilot_ii i2c_dev i2c_core i2c_boardinfo(P) fpgaflash(P) fpga gpioint gpiomgr helper nandflash adc flashinfo
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c02f2274"] (dump_stack+0x0/0x14) from [] (warn_slowpath+0x64/0x9c)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c003e9e8"] (warn_slowpath+0x0/0x9c) from [] (dev_watchdog+0x13c/0x224)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: r3:dc80c000 r2:c039ea89
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: r6:00000003 r5:c03efe3c r4:dc80c000
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: r7:c03df224 r6:f0000030 r5:00000001 r4:ffffffff
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c002a320"] (default_idle+0x0/0x5c) from [] (cpu_idle+0x40/0x5c)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c002a2c4"] (cpu_idle+0x0/0x5c) from [] (rest_init+0x58/0x6c)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: r7:c03c1d38 r6:c0024f20 r5:c0024f24 r4:c03e8628
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c02f0664"] (rest_init+0x0/0x6c) from [] (start_kernel+0x280/0x2d8)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ["c0008780"] (start_kernel+0x0/0x2d8) from [<80808034>] (0x80808034)
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: r5:c03df380 r4:00053175
Jan 12 06:54:16 ORACLESP-AK00340758 kernel: ---[ end trace 1b909de60f27e9c4 ]---
Jan 12 15:20:23 ORACLESP-AK00340758 kernel: host power off
Jan 12 15:20:23 ORACLESP-AK00340758 kernel: Host power good
Jan 12 15:20:23 ORACLESP-AK00340758 kernel: In ELXUSB20_SoftDisconnect
Jan 12 15:20:32 ORACLESP-AK00340758 kernel: ILOM UART1 IOCTL DTR bounce started.
Jan 12 15:20:33 ORACLESP-AK00340758 kernel: ILOM UART1 WORK DTR bounce stopped.

##### ilom/@persist@host_logs@host0_status.log #####

##### ilom/traces/@coredump@sp_trace@logs@CRIT.log Repeated messages filtered #####

EVTMGR CRITICAL 2016-01-05 10:55:10.659221 1216 libevtmgr_util.c:84 unknown event class: 0x00000000
EVTMGR CRITICAL 2016-01-05 10:55:10.661148 1216 libevtmgr_util.c:115 unknown event class or type: 0x00000000 / 0x00000000
LIBFRU CRITICAL 2016-01-05 10:55:10.697440 1510 dynafrud.c:397 EM disconnected

##### ilom/traces/@coredump@sp_trace@logs@GM.log Repeated messages filtered #####

GM 151 2016-01-05 17:59:56.736526 1685 version.c:34 ??truncated??
GM 41 2016-01-05 17:59:57.237356 1685 asyncio.c:945 Created LISTENER (aiop 0xa9fc8) for 'var-config-backup'
GM 41 2016-01-05 17:59:57.276411 1685 asyncio.c:957 Created SERVER (aiop 0xaa118) for 'var-config-backup'
GM 41 2016-01-05 17:59:57.296483 1685 asyncio.c:945 Created LISTENER (aiop 0xaa298) for 'dr-pdom'
GM 41 2016-01-05 17:59:57.316434 1685 asyncio.c:957 Created SERVER (aiop 0xaa3d0) for 'dr-pdom'
GM 41 2016-01-05 17:59:57.336482 1685 asyncio.c:945 Created LISTENER (aiop 0xec6f0) for 'mdstore'
GM 41 2016-01-05 17:59:57.356543 1685 asyncio.c:957 Created SERVER (aiop 0xec828) for 'mdstore'
GM 41 2016-01-05 17:59:57.366481 1685 asyncio.c:945 Created LISTENER (aiop 0xec960) for 'fma-phys-mem-service'
GM 41 2016-01-05 17:59:57.406444 1685 asyncio.c:957 Created SERVER (aiop 0xeca98) for 'fma-phys-mem-service'
GM INFO 2016-01-05 17:59:57.537104 1685 error_svc.c:1077 error_pri_init: /persist/vbsc/serlog will be created/reinitialised
GM 41 2016-01-05 17:59:57.576656 1685 asyncio.c:945 Created LISTENER (aiop 0xfcfd8) for 'fma'
GM 41 2016-01-05 17:59:57.626676 1685 asyncio.c:957 Created SERVER (aiop 0xfd120) for 'fma'
GM 41 2016-01-05 17:59:57.659685 1685 asyncio.c:945 Created LISTENER (aiop 0xfd2f8) for 'ipmi'
GM 41 2016-01-05 17:59:57.706482 1685 asyncio.c:957 Created SERVER (aiop 0xfd430) for 'ipmi'
GM 41 2016-01-05 17:59:57.837087 1790 rpc_util.c:393 Start listening for RPC msgs (UNIX server).Program ID = 0x30000004, ver = 1
GM INFO 2016-01-05 17:59:57.938513 1685 fpga_mbox.c:336 Uninstalling all mailbox interrupts
GM 41 2016-01-05 10:47:01.436587 1685 asyncio.c:548 aio_disable_asyncio: LISTENER 'dr-pdom' already disabled
GM 41 2016-01-05 10:47:01.437583 1685 eusb_skt.c:331 Disconnecting aio socket listener for "dr-pdom" LDC
GM 41 2016-01-05 10:47:01.438029 1685 asyncio.c:382 LISTENER 'dr-pdom' is already disconnected.
GM 41 2016-01-05 10:47:01.438473 1685 eusb_skt.c:336 Disconnecting aio socket server connection for "dr-pdom"
GM 41 2016-01-05 10:47:01.438918 1685 asyncio.c:382 SERVER 'dr-pdom' is already disconnected.
GM 41 2016-01-05 10:47:01.439361 1685 asyncio.c:548 aio_disable_asyncio: LISTENER 'fma-phys-mem-service' already disabled

##### ilom/console.txt Converted Text Friendly #####

2016-01-05 18:28:52.643 0:0:0:0"POST 5.3.2 2015/10/30 13:52
2016-01-05 18:28:53.985 0:0:0:0"POST Running.
2016-01-05 18:34:05.648 0:0:0:0"POST return to PROM
2016-01-05 18:34:06.180 0:0:0:0"POST Phase Complete
2016-01-05 18:34:07.245 0:0:0:0"POST Exit reason = 0
2016-01-05 18:37:26.103 0:0:0:0"POST Running.
OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.

2016-01-08 16:35:10.919 0:0:0:0"POST 5.3.2 2015/10/30 13:52
2016-01-08 16:35:12.262 0:0:0:0"POST Running.
2016-01-08 16:40:27.867 0:0:0:0"POST return to PROM
2016-01-08 16:40:28.398 0:0:0:0"POST Phase Complete
2016-01-08 16:40:29.463 0:0:0:0"POST Exit reason = 0
2016-01-08 16:43:48.419 0:0:0:0"POST Running.
2016-01-08 16:57:44.098 0:0:0:0"POST Critical region memory check
2016-01-08 17:08:42.164 0:0:0:0"POST Exit reason = 0
OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.

SunOS Release 5.11 Version 11.3 64-bit
Jan 11 11:50:22 oracle sendmail[1361]: My unqualified host name (oracle) unknown; sleeping for retry
Jan 12 15:19:02 oracle syslogd: going down on signal 15
syncing file systems... done
OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.

========== Analysis ============

If a known product issue is detected, then an analysis with bug# & related doc is presented
DD tracking early T7 faults for HW QA.

A Timeline containing data from related/older Explorers & Snapshots
*** TIMELINE - Please note that entries may be longer, & are truncated at 150 characters!!! ***

3-12010130931 16/01/14-00:19:12 Explorer/Snapshot from SR 3-12010130931
3-12010130931 16/01/13-14:37:33y Host: Solaris running
3-12010130931 16/01/13-14:34:38Q SunOS Release 5.11 Version 11.3 64-bit
3-12010130931 16/01/13-14:34:38P OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.
3-12010130931 16/01/13-14:34:38O syncing file systems... done
3-12010130931 16/01/13-14:34:38N oracle03 syslogd: going down on signal 15
3-12010130931 16/01/12-23:38:09 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW
3-12010130931 16/01/12-23:38:09 Sensors near 0 for 3 periods: /SYS/MB/CM/VCORE0_PHASE1_ADJCNT /SYS/MB/CM/VCORE0_PHASE3_ADJCNT /SYS/MB/CM/VCORE1_PHASE1_ADJCNT /SYS/MB/CM/VCORE1_
3-12010130931 16/01/12-22:52:08 Sensors near 0 for 3 periods: /SYS/MB/VREF_CPU_LOW
3-12010130931 16/01/12-22:52:08 Sensors near 0 for 3 periods: /SYS/MB/CM/VCORE0_PHASE1_ADJCNT /SYS/MB/CM/VCORE0_PHASE3_ADJCNT /SYS/MB/CM/VCORE1_PHASE1_ADJCNT /SYS/MB/CM/VCORE1_
3-12010130931 16/01/12-18:22:33z Host: Solaris running
3-12010130931 16/01/12-18:19:31M SunOS Release 5.11 Version 11.3 64-bit
3-12010130931 16/01/12-18:19:31L OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.
3-12010130931 16/01/12-18:19:31K syncing file systems... done
3-12010130931 16/01/12-18:19:31J oracle03 syslogd: going down on signal 15
3-12010130931 16/01/12-15:40:50z Host: Solaris running
3-12010130931 16/01/12-15:39:07I SunOS Release 5.11 Version 11.3 64-bit
3-12010130931 16/01/12-15:39:07H OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.
3-12010130931 16/01/12-15:22:04z Host: Solaris running
3-12010130931 16/01/12-15:19:02G SunOS Release 5.11 Version 11.3 64-bit
3-12010130931 16/01/12-15:19:02F OpenBoot 4.38.2, 478.2500 GB memory available, Serial #110783192.
3-12010130931 16/01/12-15:19:02E syncing file systems... done
3-12010130931 16/01/12-15:19:02D oracle03 syslogd: going down on signal 15
3-12010130931 16/01/11-11:09:38z Host: Solaris running

Attachments

This solution has no attachment