Troubleshooting Guide (TSG) - Ksvcreate: Process(xxxx) Creation Failed / ORA-00445: Background Process "xxxx" Did Not Start After n Seconds

Asset ID:	1-75-1379200.1
Update Date:	2018-03-13
Keywords:

Solution Type Troubleshooting Sure

Solution 1379200.1 : Troubleshooting Guide (TSG) - Ksvcreate: Process(xxxx) Creation Failed / ORA-00445: Background Process "xxxx" Did Not Start After n Seconds

Applies to:

Oracle Database - Enterprise Edition - Version 10.1.0.2 and later
Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Full Rack - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Purpose

This article can be used as a troubleshooting guide for the following messages reported in the alert.log file. Please note that the exact process type and number may be different, so this article applies for all messages in this format:

Process %s died, see its trace file
kkjcre1p: unable to spawn %s slave process

Process %s died, see its trace file
ksvcreate: Process(%s) creation failed

You may also observe messages like:

ORA-00445: background process "J000" did not start after 120 seconds

This document also lists out the known issues reported causing the ORA-445 errors.

Troubleshooting Steps

What does this message mean ?

The message indicates that we failed to spawn a new process at the Operating System level to serve the request. There are various causes for this issue. This typically occurs when there is a shortage or misconfiguration in Operating System Resources, and thereby the problem should be investigated from an OS perspective. However there are a few causes related to the Oracle Database as well.

The default 120 seconds (after which Oracle times out) can be extended dynamically (without a database restart) by setting the following event:

$ sqlplus / as sysdba
alter system set events '10281 trace name context forever, level xxx';
-- where xxxxxx is the number of seconds to timeout at.

eg: alter system set events '10281 trace name context forever, level 300';

Information from the following previously existing articles is also included in this document:

Note 790397.1 - PROCESS J000 And M000 Die
Note 561346.1 - Ksvcreate: Process(m000) Creation Failed Errors in Alert.log

OS Configuration Checks

This error may be observed due to lack of OS resources or incorrect configuration, typically memory or swap may be insufficient to spawn a new process. Please check the list below to verify the OS settings and configuration

Execute the Health Check and Validation (HCVE) script to verify the OS configuration settings

<Note 250262.1>: RDA 4 - Health Check / Validation Engine Guide

Review the resource-related suggestions and make changes accordingly. The following 2 articles may help in understanding these suggestion better:

<Note 169706.1>: Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
<Note 225349.1>: Address Windowing Extensions (AWE) or VLM on Windows Platforms
(typically on windows-system with more than 4Gb of RAM, enabling the /3GB switch in the boot.ini is required)Check the define user limitation (ulimit) settings (UNIX-only)

Check the defined user limitation(ulimit) settings (UNIX-only)

As the oracle user (or the owner of the oracle software)

# ulimit -a

Minimum values can be found in

<Note 169706.1>: Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)

Verify the SWAP space configured

Some guidelines regarding swap space configuration:

If you have between 1 and 2Gb of physical RAM, then you need to configure 1.5 times the physical RAM for swap space.
For 2 to 8Gb RAM, swap space has to be equal to the physical RAM.
For RAM more than 8Gb, swap needs to be 0.75 RAM

Check the OS Error Log and the Trace Files

You can check the OS Error Log at the time of the issue.

Note 1349613.1: How To Gather The OS Logs For Each Specific OS Platform

Also check the trace files generated at the time of the issue. Look for the 'load verage' and the memory and swap status at the time of the issue.

OS Watcher to monitor the resource usage

The previous action points typically will investigate upon OS configuration issues, however the problem may also be caused by a temporary resource spike or system overload. Please check the timings for the problem to appear in the alert.log, if it happens always around the same time, then you are dealing with a temporary resource spike. This usually occurs around times the system load is very high

OS Watcher is an Oracle tool that allows you to monitor the system from an OS perspective

<Note 301137.1>: OS Watcher User Guide
<Note 946107.1>: What Is The OSWATCHER (OSW) Effect On The Server Performance And What Are The Alternative Commands?

Issues caused by the Linux feature Address Space Layout Randomization (ASLR

This problem is reported in Redhat 5 and Oracle 11.2.0.2. You can verify whether ASLR is being used as follows:

# /sbin/sysctl -a | grep randomize
kernel.randomize_va_space = 1

If the parameter is set to any value other than 0 then ASLR is in use. Refer the document for details:

Note 1345364.1: ORA-00445: Background Process "xxxx" Did Not Start After 120 Seconds

The solution will be to disable ASLR

Checks on Oracle Database Configuration

Setting PGA_AGGREGATE_TARGET=TRUE

The parameter pga_aggregate_target is a numeric value not a boolean value and therefore must be set to a number for it to function correctly. By specifying it to a text string, we will try to convert it to a meaningful value but which may be insufficient for your environment

Solution: Properly set PGA_AGGREGATE_TARGET to a numeric value.

Setting the PRE_PAGE_SGA to TRUE or Altering SGA_SIZE with PRE_PAGE_SGA set to TRUE

PRE_PAGE_SGA instructs Oracle to read the entire SGA into active memory at instance startup. Operating system page table entries are then prebuilt for each page of the SGA. This setting can increase the amount of time necessary for instance startup, but it is likely to decrease the amount of time necessary for Oracle to reach its full performance capacity after startup. PRE_PAGE_SGA can increase the process startup duration, because every process that starts must access every page in the SGA, this can cause the PMON process to take longer to start and exceed the timeout which is by default 120 seconds causing the instance startup to fail.

Setting PRE_PAGE_SGA to TRUE can increase the process startup duration, because every process that starts must access every page in the SGA, however overhead can be significant if your system frequently creates and destroys processes by, for example, continually logging on and logging off.

Check whether PRE_PAGE_SGA is set to TRUE
--OR--
Verify the generate trace for the occurance of function: ksmprepage()

Solution: Setting PRE_PAGE_SGA to FALSE will avoid this code executing so pages are only touched as needed rather than touching every single page when the process starts. This can avoid or minimize the problem from occuring however the underlying cause is still an Operating System resource shortage

Note: The default for pre_page_sga has been changed to true in 12.1.0.2 and higher. Oracle Recommends to disable pre_page_sga for DB version lower than 12c.

Known Issues

Confirmed Code Bugs

<Bug 13036685

Closed as duplicate of Bug 12830339

ORA-00445: BACKGROUND PROCESS "XXXX" DID NOT START AFTER 120 SECONDS

<<Bug 5583049: 'Ksvcreate: Process(m000) creation failed' after Standby Database Open Read Only Multiple Times

<Note 418553.1>: 'Ksvcreate: Process(m000) creation failed' after Standby Database Open Read Only Multiple Times

<< Bug 8426816: PMON may hang cleaning up a dead process (rare)

<Note 1233079.1>:Ksvcreate: Process(M001) Creation Failed , Database hang

OS Related Bugs

Following is a list of Bugs reported which turned out to be OS specific issues.

Unpublished Bug 8336987: TB:SH:ORA-00445: BACKGROUND PROCESS "PZ99" DID NOT START AFTER 120 SECONDS
---> Platform 226 - Linux x86-64
All the time, the process is stuck in 'D' state, i.e uninterruptible state in the kernel. This is a known issue with the Linux ext3 filesystem when there is a lot of I/O activity and the filesystem has to flush out it's journal and all other I/O freezes while it is doing that. If you are willing to try it, you could change the mount options for your ext3 filesystems to include -o data=writeback and see if this problem reproduces.

Unpublished Bug 7562673: FAILED IN SPAWNING MMON SLAVES AND FLOODING OF MMON TRACES IN AQ TEST
--> Status 92 - Closed, Not a Bug
WARNING: Slow process spawn detected for OS id=4260, M000 The process exec took more than 133 secs.
CAUSE: Your oracle executable is on a remote machine. Network issues can delay loading the executable image and therefore delay the process exec. Copy the oracle image / oracle home to the instance's local hard disk.

Unpublished Bug 7377880: STARETL: ORA-00445: BACKGROUND PROCESS "M000" DID NOT START AFTER 120 SECONDS
--> Status 32 - Not a Bug
The problem appears to be that you have too much activity on your machine; the fix is to shut down some of the activity.

Unpublished Bug 9866293: GE+RAC: INSTANCE CRASH BECAUSE SMON DIED WITH ORA-00445
--> Status 92 - Closed, Not a Bug
Process startup can be time consuming on heavily loaded systems. Systems that have a high system load, or have high network latency and the image is remote, may have large fork/exec times. In this particular case it looks like ps and gdb timed out while dumping diagnostics of the spawnee. This is a good indication of high system load/network latency.

Unpublished Bug 9543620: SQL EXEC PART:ORA-00445: BACKGROUND PROCESS "P059" DID NOT START AFTER 120 SECONDS
--> Status 92 - Closed, Not a Bug
This just looks like a system load related issue. ps and gdb timing out while getting diagnostics usually means, there may be NFS latency issues, of the image or dependent libraries are remote, or a disk latency issue if they are local.

Briefly, the possible reasons addressed in these bugs are:
1. Too much activity on your machine.
2. NFS latency issues.
3. Disk latency issue (that affects I/O).
4. Network latency.

These are the four possible reasons for the ORA-445 error you are getting. None of these reasons are related to the RDBMS itself. We recommend you involve your Network, Storage and System Administrators in this investigation.

Diagnostic Informations to be Collected for Oracle Support

If none of the above steps helped in resolving the issue, please raise an SR with the Oracle Support. Ensure to upload the following files for a speedy resolution.

a. Alert Log file

b. Trace Files generated at the time of the issue

c. OS Error Log file

d. HCVE output (Refer <Note 250262.1>: RDA 4 - Health Check / Validation Engine Guide )

e. The output:

SQL> select * from v$resource_limit;

f. Please upload the OS Watcher output also, if available.

Keyword Search
ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate
ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate ksvcreate
process creation failed process creation failed process creation failed process creation failed process creation failed
process creation failed process creation failed process creation failed process creation failed process creation failed
m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000 m000
m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001 m001
j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000 j000
j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001 j001

References

<NOTE:1345364.1> - ORA-00445: Background Process "xxxx" Did Not Start After 120 Seconds
<NOTE:237481.1> - Starting Database 8.1.7 Fails with ORA-3113 and ORA-445 on Linux
<NOTE:1600807.1> - ORA-00445: background process "J000" did not start after 120 seconds
<NOTE:416244.1> - ORA-00610 And/Or "unable to spawn jobq slave process " And/Or "Process(<>) creation failed" In The Alert Log And/Or TNS-12518/ TNS-12500 In Listener Log

Attachments

This solution has no attachment