Oracle ZFS Storage Appliance: CPU Timeout Failures

Asset ID:	1-72-2266516.1
Update Date:	2018-05-18
Keywords:

Solution Type Problem Resolution Sure

Solution 2266516.1 : Oracle ZFS Storage Appliance: CPU Timeout Failures

Applies to:

Oracle ZFS Storage ZS4-4 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Problem Description: A takeover was triggered due to a CPU Timeout failure

System: Oracle ZFS Storage ZS4-4

AK version: 2013.06.05.7.0

Error message:

------------------- ------------------------------------ -------------- --------
Time UUID msgid Severity
------------------- ------------------------------------ -------------- --------
2017-05-14/00:46:09 1c2dca7a-cac2-60de-f08b-912620d76b69 SPX86A-8002-H0 Critical

Problem Status : open
Diag Engine : fdd 1.0
System
Manufacturer : Oracle Corporation
Name : SUN SERVER X4-4
Part_Number : 34023211+23+1
Serial_Number : 1553NM9003

----------------------------------------
Suspect 1 of 1
Fault class : fault.cpu.intel.timeout
Certainty : 100%
Affects : /SYS
Status : faulted

FRU
Status : faulty
Location : /SYS
Manufacturer : Oracle Corporation
Name : SUN SERVER X4-4
Part_Number : 34023211+23+1
Serial_Number : 1553NM9003
Chassis
Manufacturer : Oracle Corporation
Name : SUN SERVER X4-4
Part_Number : 34023211+23+1
Serial_Number : 1553NM9003

Description : CPU Timeout Failures.

Response : The chassis wide service-required LED is illuminated.

Action : Please refer to the associated reference document at
http://support.oracle.com/msg/SPX86A-8002-H0 for the latest
service procedures and policies regarding this diagnosis.

Current status of the CPUs on both heads:

#### node zs4-6-a
chassis-000 zs4-6-a faulted Oracle Oracle ZFS Storage ZS4-4 1553NM9003 -- system
[.......]
cpu-000 CPU 0 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-001 CPU 1 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-002 CPU 2 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-003 CPU 3 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown

#### node zs4-5-a
chassis-000 zs4-5-a ok Oracle Oracle ZFS Storage ZS4-4 1553NM9001 -- system
[.......]
cpu-000 CPU 0 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-001 CPU 1 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-002 CPU 2 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown
cpu-003 CPU 3 ok Intel Intel(r) Xeon(r) CPU E7-8895 v2 @ 2.80GHz unknown

Cause

This known Intel Microcode issue delivered as part of the SP/BIOS update for April 2017 CPU, which was backported via:

Bug 26416616 - Backport 25788973 to AK-2013-MICRO - Adjust supported ILOM/BIOS upgrade paths to

This was implemented in 2013.1.7.11 (OS8.7.11) as part of the Automatic SP/BIOS Upgrade by implementing April 2017 CPU SP/BIOS updates:

Doc ID 2280467.1 - Oracle ZFS Storage Appliance: Critical Security bug fixes for ZFSSA BIOS/ILOM (April 2017)

Solution

This is a known Intel Microcode issue where you need to 'markrepaired' the fault and plan an upgrade to 2013.1.7.11 (OS8.7.11) or higher where automatic SP/BIOS update will update to latest SP/BIOS where this is fixed.

Note: Minimum SP/BIOS is required for automatic SP/BIOS upgrade

See Doc ID 2285244.1 - Oracle ZFS Storage Appliance: Automatic upgrade of ZFSSA ILOM/BIOS Firmware in OS 8.7

IMPORTANT NOTE: It is reported that this issue is still seen on a ZS4-4 system running OS8.7.13 with the latest ILOM SP version (3.2.9.25 r116415 & BIOS: 24070300)

Attachments

This solution has no attachment