Warm Restart of SM Cards Running 64 bit GPLs may Experience Memory Corruption

Asset ID:	1-77-2190752.1
Update Date:	2017-05-12
Keywords:

Solution Type Sun Alert Sure

Solution 2190752.1 : Warm Restart of SM Cards Running 64 bit GPLs may Experience Memory Corruption

Applies to:

Oracle Communications EAGLE (Software) - Version EAGLE 46.3 and later
Information in this document applies to any platform.

Description

Warm restart of SM (Service Module) Cards running 64-bit GPLs may experience memory corruption and exhibit unpredictable behavior such as database corruption and reboots due to DB audit failures.

Occurrence

This problem may occur if following conditions are true

EAGLE is running release 46.3 software
The E5-SM8G-B SM card is running 64-bit GPLs.
Warm restart (init-card:data=persist) performed the above E5-SM8G-B SM card(s)

Below sample capture shows an E5-SM8G-B card running 64-bit GPLs. The GPL type can be determined from the GPL name. 64-bit GPLs will have the GPL name suffixed with "64"

> rept-stat-card:mode=full:loc=1317

eaglestp 16-10-03 18:18:12 EST EAGLE 46.3.0.0.1-68.27.0
CARD VERSION TYPE GPL PST SST AST
1317 138-022-000 DSM SCCP64 IS-NR Active -----
ALARM STATUS = No Alarms.
BLDC64 GPL version = 138-023-000
IMT BUS A = Conn
IMT BUS B = Disc
CLOCK A = Active
CLOCK B = Fault
CLOCK I = Idle
MBD BIP STATUS = Valid
MOTHER BOARD ID = SMXG B
DBD STATUS = Valid
DBD TYPE = None
DBD MEMORY SIZE = 8192M
HW VERIFICATION CODE = ----
CURRENT TEMPERATURE = 28C ( 83F)
PEAK TEMPERATURE: = 32C ( 90F) [16-09-21 02:02]
SCCP % OCCUP = 0%
SCCP SM DATA TYPE = DN
APPLICATION SERVICING
MFC MFC
SNM REQ STATUS = 24 hr: ---, 5 min: ---
INM REQ STATUS = 24 hr: ---, 5 min: ---
MTP3 REQ STATUS = 24 hr: ---, 5 min: ---
SFLOG REQ STATUS = 24 hr: ---, 5 min: ---
IPLNK STATUS
IPLNK IPADDR STATUS PST
A 192.168.120.13 UP IS-NR
B 192.168.121.13 UP IS-NR
DSM IP CONNECTION
PORT PST SST
A IS-NR Active
B IS-NR Active

Command Completed.
;

Below is the complete list of 64 bit GPLs that a SM card may run in EAGLE release 46.3

SCCP64
DEIR64
ENUM64
SIP64
BLDC64

Symptoms

Below are possible symptoms:

DB/RTDB corruption alarms on SM cards
Warm restart causing SM card to reboot multiple times
SM card reboots due to DB audit failures

Workaround

Avoid warm restart (do not use data=persist option with init-card command) of SM cards running 64 bit GPLs. Always cold restart (use init-card command with option data=refresh) SM cards running 64 bit GPLs to avoid possible problem occurrence. If a warm restart of SM cards running 64 bit GPLs is observed for whatever reason, then cold reboot those cards to avoid any possible problem.

Patches

Permanent fix is available in R46.3.1.0.0 and later. However SM8G-B cards already running on BLDC64 will not install the new bootloader automatically. After the Upgrade, the operator will have to explicitly perform the bootloader upgrade procedure on the SM8G-B cards of interest.

Scenario #I

If EAGLE is currently running R46.3.0.0.x and equipped with 64-bit SM8G-B cards (SM8G-B cards running BLDC64 flash GPL and SCCP64, DEIR64, SIP64 or ENUM64 application GPL), then upgrade EAGLE to R46.3.1.0.0 and run below "PROCEDURE TO MANUALLY UPDATE THE 64 BIT BOOTLOADER ON SM8G-B CARD".

Scenario #II

If EAGLE is currently running R46.3.1.0.0 or later and equipped with 64-bit SM8G-B cards (SM8G-B cards running BLDC64 flash GPL and SCCP64, DEIR64, SIP64 or ENUM64 application GPL) which were converted to 64-bit while the EAGLE was running R46.3.0.0.x, then run below "PROCEDURE TO MANUALLY UPDATE THE 64 BIT BOOTLOADER ON SM8G-B CARD".

PROCEDURE to MANUALLY UPDATE the 64 BIT BOOTLOADER on SM8G-B CARD

This procedure needs to be run only once on each card when the EAGLE is running R46.3.1.0.0 or later

Make sure EAGLE is currently running R46.3.1.0.0 or later
Inhibit SM8G-B card running 64-bit GPL and wait for the card to be back on IMT:

INH-CARD:LOC=<>
Run following command to verify the GPL. If BLDC64 GPL is reporting a GPL alarm, go to next step. Otherwise go to step 7:

REPT-STAT-GPL:LOC=<>
Run following command to update the flash GPL on card. Wait for the card the card to complete flash operation, reboot and come back on the IMT:

INIT-FLASH:CODE=APPR:LOC=<>
Run following command to verify that card is now running BLDC64 GPL approved for the release:

REPT-STAT-GPL:LOC=<>
Activate flash GPL

ACT-FLASH:LOC=<>
Run following command to update the bootloader on card and verify that “BOOTLOADER change for card xxxx SUCCESSFUL” message is getting printed:

INIT-FLASH:MODE=RPLCEBL:BITS=64:LOC=<>
Example showing expected output:

init-flash:loc=3101:mode=rplcebl:bits=64
EAGLESTP 17-04-17 01:37:26 HST EAGLE 46.4.0.0.0-69.19.0
init-flash:loc=3101:mode=rplcebl:bits=64
Command entered at terminal #21.
;
Command Accepted - Processing
EAGLESTP 17-04-17 01:37:26 HST EAGLE 46.4.0.0.0-69.19.0
BOOTLOADER change request sent to card 3101.
;
EAGLESTP 17-04-17 01:37:26 HST EAGLE 46.4.0.0.0-69.19.0
BOOTLOADER change for card 3101 SUCCESSFUL.
;
EAGLESTP 17-04-17 01:37:26 HST EAGLE 46.4.0.0.0-69.19.0
Command Completed.
;
Command Executed
Allow SM8G-B card:

ALW-CARD:LOC=<>

Scenario #III

If EAGLE is currently running R46.3.0.0.x and 32-bit SM8G-B cards (SM8G-B cards running BLMCAP flash and SCCPHC, DEIRHC, SIPHC or ENUMHC application GPL), and needs to be converted 64-bit, then do following:

Upgrade the EAGLE to R46.3.1.0.0 to first
Make sure the SM8G-B cards flashed with BLMCAP GPL approved for the release. Card should not be reporting BLMCAP GPL alarm.
Then follow the procedure documented in EAGLE documentation to perform SM8G-B Bootloader Swap and convert the card to 64-bit.

History

06-Oct-2016 - Initial draft

07-Oct-2016 - Updated based on review comments

13-Oct-2016 - Updated based on review comments

19-Oct-2016 - Published. Reviewed by: James Young, Mark Erickson, Carlos Vidales, Sudarsan Ramasamy & Michael Sams

09-May-2107 - Added procedure to update the bootloader on SM8G-B cards to get the

References

<BUG:24345772> - R46.4_SCCP_SLIC:INCORRECT BEHAVIOR OF SCCP_SLIC_ELAP CARDS WITH WARM RESTART .

Attachments

This solution has no attachment