Oracle Communications Policy Management
 

PLA-Support>Sun Systems>CommsGBU>Broadband Network Solutions>SN-SND: Tekelec Policy
 
Tools>Primary Use>Performance
 

After an upgrade of MPE/MRA blades from 11.5.x to 12.x, the following symptoms are present. 
 
1. System will start showing alarm 72904 – “Diameter too busy”  and send 3004 Diameter Error to the Gateway.
2. An increase in the system memory utilization percentage to above 80% can be seen in the KPI dashboard . G6 blades with 48GB RAM are more prone to this symptom.

In this Document

Description

Occurrence

Symptoms

Workaround

Patches

History

References

Applies to:

Description

This Alert informs OCPM customers that Oracle has found a defect in OCPM 12.x that could impact system performance.  If you recently had your Policy software upgraded to OCPM 12.x from 11.5.x, then you should read this Alert and take immediate actions.

Occurrence

If your system has recently been upgraded from OCPM 11.5.x to 12.x, they you are at risk of encountering this issue.

Symptoms

After software upgrade to OCPM 12.x, the following symptoms could be observed:

1. Alarm 72904 – “Diameter too busy” and send 3004 Diameter Error to the Gateway.
2. CMP GUI KPI Dashboard system memory utilization goes above 80% .
   
   

Workaround

Oracle has discovered the cause of high memory usage and alarms. This is caused by the existing policies that uses “Evaluate as expression”, which when evaluated could lead to Garbage Collection (GC) to pause for few secs. This pause for few secs could cause MPE threads to get hung and drive system into BUSY state and/or increase memory utilization. OCPM deployments with HP BL460 G6 blades with 48GB RAM are more prone to experience this issue.

If you experience this, Oracle provides the following work around to check if “Evaluate as expression” is being used in policies as well as how to resolve without Oracle Support involvement. Should you have an concerns with executing these steps, please open a new SR for Oracle to provide additional support.

Steps to how to identify if "Evaluated as expression" is being used:

1. Export a policy XML file from the CMP GUI
2. Open this exported XML file in a text editor
3. Search for “###EVAL###” .
   Eample:
   <Policy>
   <Name>Policy-Evaluate as Expression</Name>
   <Description></Description>
   <Id>281474977512327</Id>
   <Version>3</Version>
   <ActionValues>OACT_6,OACT_YORK_2,MACT_0</ActionValues>
   <ConditionValues>REQ_49,USER_34,REQ_49[1]</ConditionValues>
   <ConditionVariables>REQ_49_0=ENTITLEMENTS=Entitlements|REQ_49_1=ML_FUP_3072|OACT_YORK_2_0=G_Postpaid_Cenam_Services_v2 - LTE|USER_34_0=greater than or equal to=###ge###|USER_34_1=30|USER_34_2=total volume=QUOTATYPE_VOLUME|USER_34_3={User.Custom6}=###EVAL###
   User.Custom6}|REQ_49_0[1]=ENTITLEMENTS=Entitlements|REQ_49_1[1]=ML_FUP_Speed_128|OACT_6_0=policyparam.diameter.bearer.MaxRequestedBandwidthDownlink:Diameter Bearer Maximum-Requested-Bandwidth-
   L:Flow.MaxRequestedBandwidthDownlink:128000:128000;policyparam.diameter.bearer.MaxRequestedBandwidthUplink:Diameter Bearer Maximum-Requested-Bandwidth-UL:Flow.MaxRequestedBandwidthUplink:128000:128000</ConditionVariables><Analytics>false</Analytics>
   </Policy>
4. If you do find such policy lines with contains the search expression , then this confirms there are policies which are using “Evaluate as expression” and you should proceed to step 5. If not found, then you should open a new SR for Oracle to provide additional support.
5. Make a list of all policy names which were found while looking for “##EVAL##” in policy export XML file.
6. In CMP GUI, access those policies from “Policy Library ” under “Policy Management” and click “modify".
7. Identify the condition or action which is using “Evaluate as expression” in that policy.  This can be done by checking the XML and then looking at the corresponding line in the policy on the CMP GUI.
8. Once clicked on that line you would see a pop-up showing “Evaluate as expression” being checked mark. Evaluate if that condition or action is actually making use of “Evaluate as expression” to compute some regex or arithmetic expression.
   1. If the identified condition and/or action is NOT doing any computation , then uncheck the “Evaluate as expression” option and select “ok” and then click “Finish” to save the modified policy
   2. If the identified condition and/or action is actually doing some computation, then that policy / policy logic needs to be modified without using “Evaluate as expression” to achieve the same results.
9. Repeat steps 6 - 8 until all policies identified in Step 5 have been validated, corrected and redeployed.
10. Confirm system is free of symptoms reported in this Alert. If symptoms continue, then please open a new SR for Oracle to provide additional support.

Patches

Oracle has identified root cause for this issue. It is reported on BUGID 26171179 and the issue is fixed in Policy release version 12.2.2 and 12.3. 12.3 which will be available on 11-July-2017.

History

14-JUN-2017 - Original Document

References

This solution has no attachment