Integrated Software for BNS Hardware (UDR)
 
Oracle Communications Performance Intelligence Center (PIC) Hardware
 
Integrated Software for BNS Hardware (DSR)
 
Integrated Software for BNS Hardware (Policy)
 

PLA-Support>Sun Systems>CommsGBU>Global Signaling Solutions>SN-SND: Tekelec DSR
 

Potential to affect ProLiant servers with Smart Array Px2x Controllers operating firmware versions lower than 8.32 per HPE advisory a00029265.

In this Document

Description

Occurrence

Symptoms

Workaround

Patches

History

Applies to:

Description

Applies to HP Blade and Rack-Mount Servers using Smart Array Px2x Controllers operating firmware release 8.31 or previous.  Hardware includes HP bl460c and dl380p (which use HP Smart Array p220i and p420i Controllers respectively) commonly used for Oracle Communications software applications including DSR, UDR, Policy PCRF, and others.

Some reports have arisen where an Oracle Communications application operating on an HP server abruptly stops processing data.  The application raises numerous alarms based on the server role.  Active intervention is required to recover server operation.

The issue is traced to a condition outlined in HPE Advisory a00029265 which recommends upgrading to Smart Array Firmware Version 8.32 or higher to prevent system unresponsiveness.  It has been identified as a "livelock" condition where the RAID Stack thread is polling for completion to be returned by the base code firmware.  In this situation no writes can be performed to disk, which causes the server OS functionality to slow down, application issues may appear, and eventually OS can hang.  When this arises, the system can stop responding.

Occurrence

Observed appearance of this condition has been rare, but depending on the role of the affected server and path to recovery the impact can be significant.

Symptoms

Based on the role of the server, alarms of many types may be raised.  Servers participating in an active/standby or multi-active group may or may not fail over responsibilities to other servers, depending on which functions are able to respond.

Viewing the Integrated Management Log (IML) via the c7000 Onboard Administrator (OA) for blades or directly in the server's iLO interface, the following Critical level entry will be observed (where x is the slot number):

  Drive Array Controller Failure (Slot x)

Executing 'syscheck' on the server--if possible--may* return errors in module class "disk" similar to the following:

 hpdisk: FAILURE:: MAJOR::3000000200000000 -- The hpacucliStatus utility needs intervention.
 hpdisk: FAILURE:: Failure message: The HP disk status is stale, and server has been up longer than 600
 smart: FAILURE:: MINOR::5000000000040000 -- Platform Health Check Failure
 smart: FAILURE:: Error: Cannot open lock file: /var/TKLC/log/smartd/lock.

     *note that any reported disk module errors may not match these exactly, as it depends on when the condition arose.

Workaround

If the condition is detected, recovery may be achieved by rebooting the server via the iLO interface.  If recovery fails, the server should be replaced.

To prevent this issue from arising, Oracle recommends upgrading HPE Smart Array Controller firmware to version 8.32 or higher per the HPE Advisory. 

Patches

For customers who have purchased hardware through Oracle (or previously Tekelec), Oracle Communications has updated Firmware Upgrade Pack (FUP) releases 2.2.11 and 2.2.12 with an addendum to include instructions to upgrade HPE Smart Array Controller firmware to version 8.32.  The updated Smart Array Controller firmware is available as an additional download to the existing FUP bundle for those releases.  Oracle recommends updating the HPE Smart Array Controller firmware as soon as possible to avoid potentially encountering this condition.

For customers who have sourced HPE hardware separately, Oracle recommends upgrading Smart Array Controller firmware as advised by HPE.
Refer to the HPE Advisory for information on how to download and apply the firmware update.

HPE Advisory a0002926
https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=null&docLocale=en_US&docId=emr_na-a00029265en_us

History

 23-MAY-2018 - Initial Publication
 29-MAY-2018 - Removed mention of FUP2.2.13 (unreleased), add recommendation to update HPE Smart Array Controller FW.

This solution has no attachment