FS System: How to Triage an FS1 Pilot or Controller Issue Using the ILOM Snapshot Bundle

Asset ID:	1-79-2200386.1
Update Date:	2018-04-30
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2200386.1 : FS System: How to Triage an FS1 Pilot or Controller Issue Using the ILOM Snapshot Bundle

Applies to:

Oracle FS1-2 Flash Storage System - Version 6.2 to 6.2 [Release 6.2]
Information in this document applies to any platform.

Purpose

This document will describe the contents of the tar file generated from the ilomsnapshot.pl utility. Support personnel can use it in locating relevant data quicker.

Scope

This document does not cover the execution of the utility. For information on that subject, refer to KM Document 1963071.1 FS System: How to Collect an ILOM Snapshot from the Pilot or Controller. The ilomsnapshot.pl utility only works reliably in R6.2.9 and beyond. Initially this document will list the files found in the tar file as well as the known descriptions of same. As more details are understood, they will added - feedback is welcome.

Details

ILOM Snapshot Filename:

Once the ILOM snapshot logs are generated their filenames will be created with the following format:

ORACLESP-<Serial_Number>_<Serial_Number>_YYYY-MM-DDTHH-MM-SS.zip

with the Serial Number being that of the individual component (Pilot or Controller) that the snapshot was gathered on. The snapshot files themselves can be found on the Pilot from which the snapshot was executed in the following locations and would need to be downloaded to another system so they can be attached to a Service Request:

Active Pilot (self) = /var/tmp/snapshot/self
Standby Pilot (buddy) = /var/tmp/snapshot/buddy
Executing Controller (self) = /var/images/tds/slammer<WWN of executing Controller>/snapshot/snapshot_self
Non-Executing Controller (buddy) = /var/images/tds/slammer<WWN of executing Controller>/snapshot/snapshot_buddy

Files Contained Within the Directories:

For easy identification, the most useful files are highlighted in bold.

Note: due to formatting issues, these files are best viewed using Notepad++ or the cores web interface.

fma directory files:
- @persist@faultdiags@ereports.log = error reports
- @persist@faultdiags@faults.log = detailed information on system faults and status
- @usr@local@bin@fmadm_faulty.out = output of fmadm faulty command.
- @usr@local@bin@fmdump_-ev.out = verbose fault management dump of events
- @usr@local@bin@fmdump_-v.out = verbose dump of faults
- @usr@local@bin@fmstat.out = rules engine statistics
fruid directory files (X5-2 Pilot only):
- @persist@frutemp@fru*.dat = data files (*= integers) used by fruimage.xml
- @persist@frutemp@fruimage.xml = maps *dat files to their containers
- @usr@local@bin@capifruidentities.out = maps containers to part and serial numbers as applicable
- @usr@local@bin@fruimagedump_-b.out = log of fruid collection (also shows mapping between containers and *dat files)
- @usr@local@bin@serial_number_check.out = empty??
- @usr@local@bin@showpsnc.out = Product Serial Number Containers output.
- @usr@local@bin@showsvcid.out = Service ID output (Manufacturer, Name, Part #, Serial #)
hwdiag directory files (X5-2 Pilot only):
- @usr@local@bin@hwdiag_cpld_bscan_DBP_FPGA_all.out = shows boundary scan (bscan) data for the Disk Back Plane Field Programmable Gate Arrays (FPGAs)
- @usr@local@bin@hwdiag_cpld_bscan_PWRCTL_FPGA_all.out = shows bscan data for Power FPGAs
- @usr@local@bin@hwdiag_cpld_vr_check_PWRCTL_FPGA.out= shows status/condition of Complex Programmable Logic Devices (CPLDs)
- @usr@local@bin@hwdiag_fan_get_all.out = shows information about motherboard fans
- @usr@local@bin@hwdiag_gpio_get_all.out = General Purpose IO information
- @usr@local@bin@hwdiag_io_error_all.out = shows information about PCIe register status
- @usr@local@bin@hwdiag_io_nvme_test.out = NVME drive information (not applicable to FS1-2 Pilots)
- @usr@local@bin@hwdiag_pci_info_all.out = shows information about PCIe probes for each CPU
- @usr@local@bin@hwdiag_power_info_all.out = Power Supply information (type, revision, voltages, current, temperatures and fan speeds)
- @usr@local@bin@hwdiag_system_fabric_test_all.out = CPU test results
- @usr@local@bin@hwdiag_system_info.out = System information (CPUs, Memory, Disks, Network Interfaces)
- @usr@local@bin@hwdiag_system_inventory.out = System details (CPUs, DIMMs, PCIes, Power Supplies)
- @usr@local@bin@hwdiag_temp_get_all.out = Current temperatures of various locations (Power Supplies, PCIe slots etc)
- @usr@local@bin@hwdiag_-v_i2c_test_all.out = Test results of I2C devices
ilom directory files:
- bbr/
- conf/
- statistics/
- traces/
- @etc@versions = detailed version information about ILOM/SP (firmware, platform, filesystem)
- @persist@fips.conf = FIPS enabled/disabled (default = disabled)
- @persist@host_debug_err.log = detailed information of hardware failures
- @persist@hostconsole.log = shows output of host (ILOM) console
- @persist@hostconsole.log.timestamp
- @persist@logmgr.log
- @persist@logmgr.log.1
- @persist@logmgr_audit.log
- @persist@logmgr_audit.log.1
- @persist@logmgr_sdm.log
- @persist@logmgr_sdm.log.1
- @persist@pod_db@cpu0.tcontrol
- @persist@pod_db@cpu1.tcontrol
- @persist@pod_db@cpupower = CPU power usage in watts
- @persist@pod_db@dbobj_store.xml
- @persist@pod_db@dbobj_store.xml~
- @persist@pod_db@fault_db.xml
- @persist@pod_db@mincpupower
- @persist@pod_db@ozone_db.xml
- @persist@pod_db@ozone_db.xml~
- @persist@servicetag.xml = shows basic system information
- @persist@spd_cache@SYS_MB_P0_D0.spd
- @persist@spd_cache@SYS_MB_P1_D0.spd
- @tmp@fips.oper = FIPS operational mode (default = disabled)
- @usr@local@bin@collect_properties.out = output from various "show properties" commands.
- @usr@local@bin@featurecheck_-show_features.out
- @usr@local@bin@featurecheck_-show_modules.out
- @usr@local@bin@invcachectl_summary.out
- @usr@local@bin@spshexec_show_-script_@System@Log@list.out = output of show /System/Log/list
- @usr@local@bin@spshexec_show_-script_@X@logs@audit@list.out = output of show /SP/logs/audit/list
- @usr@local@bin@spshexec_show_-script_@X@logs@event@list.out = output of show /SP/logs/event/list
- @usr@local@bin@spshexec_show_@SP@bootlist.out = output of show /SP/bootlist
- @usr@local@bin@spshexec_show_faulty.out = output of fmadm shell's show faulty
- @usr@local@bin@spshexec_version.out = ILOM details (version, password default?, hostname)
- @usr@local@bin@sysstatectl_summary.out
ipmi directory files:
- @bin@rm_-f_@dev@shm@sdr.raw.out
- @bin@rm_-f_@dev@shm@sel.raw.out
- @dev@shm@sdr.raw
- @dev@shm@sel.raw
- @usr@local@bin@ipmiint_bmc_info.out
- @usr@local@bin@ipmiint_chassis_restart_cause.out = shows reason for a chassis restart
- @usr@local@bin@ipmiint_chassis_status.out = provides fault status of chassis
- @usr@local@bin@ipmiint_fru_print.out = provides detailed information about system frus
- @usr@local@bin@ipmiint_lan_print.out = provides ILOM network information
- @usr@local@bin@ipmiint_pef_info.out
- @usr@local@bin@ipmiint_pef_list.out
- @usr@local@bin@ipmiint_pef_policy.out
- @usr@local@bin@ipmiint_pef_status.out
- @usr@local@bin@ipmiint_sdr_dump_@dev@shm@sdr.raw.out
- @usr@local@bin@ipmiint_sdr_elist_all.out = Sensor Data Repository (SDR) details (device Present/Absent)
- @usr@local@bin@ipmiint_sdr_info.out
- @usr@local@bin@ipmiint_sdr_list_all.out
- @usr@local@bin@ipmiint_sel_elist.out
- @usr@local@bin@ipmiint_sel_info.out
- @usr@local@bin@ipmiint_sel_writeraw_@dev@shm@sel.raw.out
- @usr@local@bin@ipmiint_sensor_list.out
- @usr@local@bin@ipmiint_sunoem_led_get.out = LED status (OFF/ON/na)
spos_info directory files:
- net/
- proc_fd/
- proc_status/
- @bin@df_-k.out
- @bin@ps_-el.out
- @proc@cpuinfo
- @proc@devices
- @proc@interrupts
- @proc@loadavg
- @proc@meminfo
- @proc@mounts
- @proc@mtd
- @proc@partitions
- @proc@slabinfo
- @proc@stat
- @proc@sysvipc@shm
- @proc@uptime = contains SP uptime in seconds (first number)
- @usr@bin@free.out
- @usr@bin@top_-bn_2.out
spos_logs directory files:
- @bin@ls_-lR_@var@log.out
- @var@log@daemon_critical.log
- @var@log@dmesg
- @var@log@ealertd.log
- @var@log@htsignon
- @var@log@hwdiag_i2c_test.log
- @var@log@ipv6_proxy.log
- @var@log@libfishwrap.log
- @var@log@lumain.log
- @var@log@lumain.log.1
- @var@log@messages
- @var@log@messages.1
- @var@log@networking
- @var@log@snmpd.err.log
- @var@log@snmpd.log
- @var@log@sppost.log
- @var@log@usrmgt.log

Attachments