ODA oakcli show network outputs "Gathering Statistics..." on version 12.1.2.0 or lower

Asset ID:	1-72-2283599.1
Update Date:	2017-07-17
Keywords:

Solution Type Problem Resolution Sure

Solution 2283599.1 : ODA oakcli show network outputs "Gathering Statistics..." on version 12.1.2.0 or lower

Applies to:

Oracle Database Appliance X4-2 - Version All Versions and later
Information in this document applies to any platform.
Note that symptom mentioned in this note is applicable for ODA versions 12.1.2.0.0 or lower

Symptoms

On one of the ODA node, command oakcli show network outputs "Gathering Statistics..." instead of the actual network details of ODA.

[root@xyz~]# oakcli show network
Gathering Statistics...
[root@xyz~]#

Cause

The affected node was running on low system *free* memory and the load on the node was high

MemTotal: 32750388 kB
MemFree: 850652 kB <<< only 800 MB left in the system as free memory
SwapFree: 14517940 kB
HugePages_Free: 6583

top - 08:43:25 up 34 days, 15:01, 1 user, load average: 8.93, 7.01, 4.08 <<<
top - 08:43:45 up 34 days, 15:01, 1 user, load average: 10.82, 7.56, 4.34
top - 08:44:06 up 34 days, 15:02, 1 user, load average: 18.42, 9.50, 5.05 <<< this machine was configured with 4 CPU cores, load average is high
top - 08:44:16 up 34 days, 15:02, 1 user, load average: 21.12, 10.38, 5.38 <<<
top - 08:44:48 up 34 days, 15:02, 1 user, load average: 28.34, 13.15, 6.46 <<<

Checking further, we could see that oakd process leaks huge chunk of anonymous memory making the system to run low on memory. pmap output on oakd process confirms the same:

== pmap output ==

+ memory value in KB

anon 26768488 <<< tops the list, equal to 25.5GB
/opt/oracle/oak/lib/libclntsh.so.12.1 48756
/opt/oracle/oak/lib/libhasgen12.so 16316
/opt/oracle/oak/lib/libnnz12.so 7204
/opt/oracle/oak/lib/libclntshcore.so.12.1 5560
/opt/oracle/oak/bin/oakd 4828
/opt/oracle/oak/lib/libipc1.so 3556
/opt/oracle/oak/lib/libasmclntsh12.so 3432
/lib64/libc-2.5.so 3408
/opt/oracle/oak/lib/libocr12.so 3116
/opt/oracle/oak/lib/libskgxp12.so 3076
/usr/lib64/libstdc++.so.6.0.8 3000
/opt/oracle/oak/lib/libocrb12.so 2984
/opt/oracle/oak/lib/libcell12.so 2788
/lib64/libm-2.5.so 2572

Solution

oakd process leaking memory is a known bug and the fix for the same is available starting ODA/OAK version 12.1.2.1.0. As a work-around restart the oak, the memory will get released. Even after restart of oak, the command "oakcli show network" still shows output as "Gathering Statistics...", then try to restart Oracle Clusterware or reboot the node. The issue is primarily due to high load and low free memory on the server.

# oakcli restart oak
or
# crcstl stop crs
# crsctl start crs

References

<BUG:20408156> - OAKD CONSUMES HIGH MEMORY
<BUG:25772598> - OAKD IS CONSUMING MORE THAN 7.5 GB OF RAM
<BUG:20132837> - LNX64-121-CMT: 'OAKCLI SHOW COOLING' DOESN'T WORK: GATHERING STATISTICS...

Attachments

This solution has no attachment