12c: Warning Message About Heavy Swapping Observation and ORA-700 [kskvmstatact: excessive swapping observed]

Asset ID:	1-72-1919850.1
Update Date:	2018-04-26
Keywords:

Solution Type Problem Resolution Sure

Solution 1919850.1 : 12c: Warning Message About Heavy Swapping Observation and ORA-700 [kskvmstatact: excessive swapping observed]

Applies to:

Oracle SuperCluster M7 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.2.0.1 [Release 12.1 to 12.2]
Information in this document applies to any platform.

Symptoms

Alert log reports frequent information about memory being swapped like below:

...
WARNING: Heavy swapping observed on system in last 5 mins.
pct of memory swapped in [2.70%] pct of memory swapped out [0.00%].
Please make sure there is no memory pressure and the SGA and PGA
are configured correctly. Look at DBRM trace file for more details.
Errors in file /orasw01/oracle/diag/rdbms/elfadev/ELFADEV/trace/ELFADEV_dbrm_21092.trc (incident=16909):
ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /orasw01/oracle/diag/rdbms/elfadev/ELFADEV/incident/incdir_16909/ELFADEV_dbrm_21092_i16909.trc
...

The sum of the pct memory swapped in and swapped out always appears to be higher that 2%.

The DBRM trace file shows information similar to:

Heavy swapping observed in last 5 mins:    [pct of total memory][bytes]
total swpin [ 3.60%][599496K], total swpout [ 0.00%][0K]
vm stats captured every 30 secs for last 5 mins:
swpin:                 swpout:  
[ 0.07%][     12416K]  [ 0.00%][         0K]
[ 0.38%][     63712K]  [ 0.00%][         0K]
[ 0.05%][      9432K]  [ 0.00%][         0K]
[ 0.81%][    135488K]  [ 0.00%][         0K]
[ 0.35%][     59032K]  [ 0.00%][         0K]
[ 0.22%][     37744K]  [ 0.00%][         0K]
[ 0.35%][     58544K]  [ 0.00%][         0K]
[ 0.62%][    104592K]  [ 0.00%][         0K]
[ 0.56%][     93368K]  [ 0.00%][         0K]
[ 0.15%][     25168K]  [ 0.00%][         0K]
Heavy swapping observed in last 5 mins:    [pct of total memory][bytes]

The call stack trace is similar to

----- Call Stack Trace -----

... kskvmstatact kskdbrmtoutact ksbcti ksbabs ksbrdp opirip opidrv sou2o ...

Changes

Upgraded to version 12.1.

Cause

Until 11.2.0.3 we don't write any warning in the alert.log.

For 11.2.0.3 and 11.2.0.4, a warning for swapping is written to the alert.log as implemented via unpublished
Bug 10220118 - NEED TO ALERT WHEN SYSTEM IS CLOSE TO PAGING

Now with 12.1.0.X, we write the warning and add more diagnostic information with the ORA-700 error if the total swap in + swap out percentage is over a certain threshold (2%).
This problem was discussed in unpublished
Bug 19474888 - ORA-700 [KSKVMSTATACT: EXCESSIVE SWAPPING OBSERVED]
closed as not a bug.
Both the alert log message and ORA-700 are expected behavior for 12c.

Solution

Given both the alert log message and ORA-700 are expected behavior for 12c, no action is required.

An enhancement request (ER) was created to change the threshold in future releases in
<Bug 19495842> - CHANGE THRESHOLD IN THE SWAP WARNING IN THE ALERT_<SID>.LOG

Note that there is no fix for that ER yet, so there is currently no way for the user to adjust the warning level.

There are some reports that the below workaround is not functioning in OEM 13.2. So made it as internal and can be provided by support in case off need to OEM versions under 13.2

Note:

To suppress ORA-700 [kskvmstatact: excessive swapping observed] in OEM, use the steps below:
a. change the threshold of "Generic Alert Log Error" metric in "db alert log" metric group from "ORA-0*(600?|7445|4[0-9][0-9][0-9])[^0-9]"
To: ORA-
b. Click on the multi pencil icon in front of "Generic Alert Log Error"
c. Change the value of "Alert Log Filter Expression" from ".*ORA-0*(54|1142|1146)\D.*"
To: ORA-00700.*kskvmstatact: excessive swapping observed.*

- Make sure to apply the changes to "Generic Alert Log Error" metric, under"db alert log" metric group, not any other "Generic Alert Log Error" under any other metric group
- By applying these changes, you will receive any ORA error alert without suppressing any alert except only: ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed]

References

<BUG:19474888> - ORA-700 [KSKVMSTATACT: EXCESSIVE SWAPPING OBSERVED]
<NOTE:1452790.1> - Warning: Heavy Swapping Observed on System after Upgrade to 11.2.0.3

Attachments

This solution has no attachment