Excessive Memory Consumption by cmha Process Leading to LSMS System Freeze

Asset ID:	1-77-2331906.1
Update Date:	2018-03-26
Keywords:

Solution Type Sun Alert Sure

Solution 2331906.1 : Excessive Memory Consumption by cmha Process Leading to LSMS System Freeze

Applies to:

Oracle Communications EAGLE (Hardware) - Version LSMS 13.2 and later
Information in this document applies to any platform.

Description

The cmha process (used by the system for high availability functionality) is consuming memory excessively and failing to release the allocated memory. This consumption of memory is causing the systems to fail once the pool of available memory is exhausted. As the system in this condition will not have available ram, the failure will likely prevent the failover from properly operating.

Bug 26671331 is open to address this issue in the software. The targeted fix is LSMS 13.3 scheduled for Jan 2018, and a Patch for LSMS 13.2.1 is available in MOS [Patch 27415594: Patch for LSMS 13.2.1.0.0-132.23.0]

Occurrence

The cmha is consuming additional memory at a fairly consistent rate. From the start of the cmha process, the application will operate for approximately 150 days before the system will run out of available memory.

Symptoms

The symptoms of a system that has exhausted the memory pool varies depending upon which application is requesting allocation of memory. The most efficient method available at this moment, to determine current system status, is to perform the checks in the workaround section below.

Workaround

Determining Present Memory Usage of cmha

The following steps will be executed to ascertain if a Maintenance window will be required to restart the cmha process.

1. On the Standby and Active LSMS, take output of the following command:

ps aux| grep "cmha"| grep -v grep | awk '{print $4}'

2. If the output is greater than 40 for any server, arrange for a Maintenance Window to restart the processes for cmha.

Restarting the cmha Processes

The following procedure should be run in the Maintenance Window for the servers where output from the step 2 above was greater than 40.

1. On the standby LSMS, take output of the following command. Note down the percentage of memory consumed by “cmha” process.

[root@lsmssec ~]# ps aux| grep "cmha"| grep -v grep | awk '{print $4}'

Also note down the output of the command “free”.

2. Stop the NPACAgents and Eagle Agents running in the Active LSMS server.

3. On the Standby LSMS, run stopNode and then startNode. Use “su – lsmsmgr” -> “Maintenance” -> “Stop Node” / “Start Node”.

4. After the Standby node is started, take output of the following command. Note down the amount of memory consumed by “cmha” process.
[root@lsmssec ~]# ps aux| grep "cmha"| grep -v grep | awk '{print $4}'

Also note down the output of the command “free”.

5. See that memory is freed after the stopNode and startNode of the Standby LSMS server.

The memory consumed by cmha will be less than 5%, it will be mostly 0% at the beginning. The memory consumed by the system (2nd row, second column of the free output) will be significantly less than 8 GB i.e. it will be less than 2 GB.

In the following output of free: The used memory is in the second row (-/+buffers/cache), second column (1430912). The value here is 1430912 KB i.e. 1.43 GB.

[epapdev@inde5epap1d1 ~]$ free
total used free shared buffers cached
Mem: 8050280 7723756 326524 0 780216 5512628
-/+ buffers/cache: 1430912 6619368
Swap: 2048184 0 2048184

6. Start the NPACAgents and Eagle Agents in the Active LSMS server. Verify both servers are running and functioning normally.

7. Switchover role i.e. Make the current Standby as Active, Active will become Standby. To make this happen, run the following command on the Standby LSMS server (This example assumes lsmssec is currently standby):

[root@lsmssec ~]# hastatus; ssh mate hastatus
[root@lsmssec ~]# /usr/TKLC/plat/sbin/hafailover --goactive
[root@lsmssec ~]# hastatus; ssh mate hastatus

8. Verify activity switchover was successful and that both servers are functioning normally.

9. Repeat the steps 1-6 so that the new Standby (previous Active) also frees its cmha occupied memory.

Note: After the activity, the lsmspri and lsmssec will have their role changed as far as Active and Standby role is concerned.

History

12-08-2017 Initial Publication

Attachments

This solution has no attachment