Provisioning Not Working And DataAccessServer Crashing Intermittently. WebCI/BlueCli Not Accessible.

Asset ID:	1-71-2165435.1
Update Date:	2017-06-20
Keywords:

Solution Type Technical Instruction Sure

Solution 2165435.1 : Provisioning Not Working And DataAccessServer Crashing Intermittently. WebCI/BlueCli Not Accessible.

Applies to:

Oracle Communications Subscriber Data Management (SDM) - Version SDM 9.3 to SDM 9.3 [Release SDM 9.0]
Tekelec

Goal

This document provides some information when provisioning is not working, the DataAccessServer is crashing intermittently and WebCI/BlueCli is not accessible.

Solution

Our investigation finds the above events were attributed to one or more of the following causes:

1. VOLTE kick-off increased traffic - evidence that Sh traffic 10minAvg >10K and peaks are almost 12K. SPR is running at its max supported performance. Running at such high capacity negatively impacts system fault tolerance.
2. Known bugs in SPR 9.3.1 patch 8/10 were exposed due to increased traffic. This bugs have been corrected in SPR 9.3.1 patch13. Sites were upgraded to Patch level 13 during restoration.
3. SPR had wrong HW configuration – HP BL460c G8 with 64GB RAM deployed on BE and HP BL460c G8 with 192GB deployed on FE. SPR specs list BE should be 192GB. Hardware swap was implemented to correct HP BL460c G8 hardware configuration issues.
4. SPR supports maximum of 100 TPS on REST. VF-TK Provisioning system does not throttle REST commands to 100/sec. There are IPTABLES rules implemented on SPR BE servers for throttling. After servers were rebuilt, prior throttling method did not survive rebuilds or upgrades. Oracle was not aware of IPTABLES configuration as they engaged with VF-TK. Customer reapplied IPTABLES rules to set limits for REST TPS.
5. SPR HP BL460c G8 hardware had older firmware that had known bugs. The recommended FW release was FUP > 2.2.5. Site had FUP 2.2.4. FUP 2.2.9 was applied to all HP BL460c G8 hardware.

Attachments

This solution has no attachment