故障排除指南(TSG) Ksvcreate进程(xxxx)创建失败ORA-00445后台进程在n秒后没有启动

原创 _ 云和恩墨 2022-10-21

1523

一、报错日志类型

Process %s died, see its trace file
kkjcre1p: unable to spawn %s secondary process

Process %s died, see its trace file
ksvcreate: Process(%s) creation failed

ORA-00445: background process "J000" did not start after 120 seconds

二、故障排除步骤

2.1、消息代表什么

该消息表明，我们未能在操作系统级别生成一个新进程来满足请求。这个问题有多种原因。当操作系统资源短缺或配置错误时，通常会出现这种情况，因此应该从操作系统的角度来调查问题。但是，也有一些原因与Oracle数据库有关。

通过设置以下事件，可以动态延长默认的120秒(此后Oracle超时)(无需重新启动数据库):

alter system set events“10281 trace name context forever，level XXX”；
-其中xxxxxx是超时的秒数。
alter system set events ' 10281 trace name context forever，level 300 '；

2.2、操作系统配置检查

出现此错误的原因可能是缺少操作系统资源或配置不正确，通常内存或交换空间不足以生成新进程。请检查以下列表以验证操作系统设置和配置。

2.2.1、执行运行状况检查和验证(HCVE)脚本来验证操作系统配置设置

[oracle@19c01 rda]$ ./rda.sh -T hcve
Processing HCVE tests ...
Available Pre-Installation Rule Sets:
   1.  Oracle Database 10g R1 (10.1.0) Preinstall (Linux)
   2.  Oracle Database 10g R2 (10.2.0) Preinstall (Linux)
   3.  Oracle Database 11g R1 (11.1) Preinstall (Linux)
   4.  Oracle Database 11g R2 (11.2.0) Preinstall (Linux)
   5.  Oracle Database 12c R1 (12.1.0) Preinstallation (Linux)
   6.  Oracle Database 12c R2 (12.2.0) Preinstallation (Linux)
   7.  Oracle Database 18c Preinstallation (Linux)
   8.  Oracle Database 19c Preinstallation (Linux)
   9.  Oracle Identity and Access Management PreInstall Check: Oracle Identity
       and Access Management 11g Release 2 (11.1.2) Linux
  10.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 11g Release 2
       (11.1.2.4) Linux
  11.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 12c (12.1.3)
       Linux
  12.  OAS PreInstall Check: Application Server 10g R2 (10.1.2) Linux
  13.  OAS PreInstall Check: Application Server 10g R3 (10.1.3) Linux
  14.  OFM PreInstall Check: Oracle Fusion Middleware 11g R1 (11.1.1) Linux
  15.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.1.3) Linux
  16.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.2.1.3.0) Linux
  17.  Oracle Forms and Reports PreInstall Check: Oracle Forms and Reports 11g
       Release 2 (11.1.2) Linux
  18.  Portal PreInstall Check: Oracle Portal Generic
  19.  IDM PreInstall Check: Identity Management 10g (10.1.4) Linux
  20.  BIEE PreInstall Check: Business Intelligence Enterprise Edition 11g
       (11.1.1) Linux
  21.  EPM PreInstall Check: Enterprise Performance Management Server (11.1.2)
       Generic
  22.  Oracle Enterprise Manager Cloud Control PreInstall Check: Oracle
       Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4) Linux
  23.  Oracle E-Business Suite Release 11i (11.5.10) Preinstall (Linux x86 and
       x86_64)
  24.  Oracle E-Business Suite Release 12 (12.1.1) Preinstall (Linux x86 and
       x86_64)
  25.  Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)

Available Post-Installation Rule Sets:
  26.  RAC 10G DB and OS Best Practices (Linux)
  27.  Data Guard Postinstall (Generic)
  28.  WLS PostInstall Check: WebLogic Server 11g (10.3.x) Generic
  29.  WLS PostInstall Check: WebLogic Server 12c (12.x) Generic
  30.  Portal PostInstall Check: Oracle Portal Generic
  31.  OC4J PostInstall Check: Oracle Containers for J2EE 10g (10.1.x) Generic
  32.  SOA PostInstall Check: Service-Oriented Architecture 11g and Later
       Generic
  33.  OSB PostInstall Check: Service Bus 11g and Later Generic
  34.  Oracle Forms 11g Post Installation (Generic)
  35.  Oracle Enterprise Manager Agent 12c Post Installation (Generic)
  36.  Oracle Management Server 12c Post Installation (Generic)
  37.  Network Charging and Control Database Post Installation (Generic)

Enter the HCVE rule set number or 0 to cancel the test
Press Return to accept the default (0)
> 8

Performing HCVE checks ...
Enter value for < Planned ORACLE_HOME location >
Press Return to accept the default (/u01/app/oracle/product/19.3.0/dbhome_1)
>


Test "Oracle Database 19c Preinstallation (Linux)" executed at 20-Oct-2022 16:58:51

Test Results
~~~~~~~~~~~~
(Note : Current Health Check rulesets do not have logic to support checks against Oracle Cloud or Exadata environments. If your product is installed in such an environment some test results will be erroneous. Refer instead to the product documentation rather than rely on the data below.)

ID     NAME                 RESULT  VALUE
====== ==================== ======= ==========================================
A00100 OS Certified?        FAILED  Not certified [Oracle Linux version]
A01020 User in /etc/passwd? PASSED  userOK
A01040 Group in /etc/group? PASSED  GroupOK
A01050 Enter ORACLE_HOME    RECORD  /u01/app/oracle/product/19.3.0/dbhome_1
A01060 ORACLE_HOME Valid?   PASSED  OHexists
A01070 O_H Permissions OK?  PASSED  CorrectPerms
A01410 oraInventory Permiss PASSED  oraInventoryOK
A01420 Other OUI Up?        PASSED  NoOtherOUI
A01430 Got Software Tools?  FAILED  LdElsewhere
A01440 Other O_Hs in PATH?  FAILED  OratabEntryInPath
A02010 Umask Set to 022?    PASSED  UmaskOK
A02030 Limits Processes     PASSED  Adequate
A02040 Limits Stacksize     PASSED  Adequate
A02050 Limits Descriptors   PASSED  Adequate
A02100 LDLIBRARYPATH Unset? FAILED  IsSet
A02170 JAVA_HOME Unset?     PASSED  UnSet
A03100 RAM (in MB)          PASSED  7641
A02210 Kernel Parameters OK PASSED  KernelOK
A02300 Tainted Kernel?      PASSED  NotVerifiable
A03010 Temp Adequate?       PASSED  TempSpaceOK
A03020 Disk Space OK?       PASSED  DiskSpaceOK
A03050 Swap (in MB)         RECORD  8191
A03100 RAM (in MB)          PASSED  7641
A03150 SwapToRam OK?        PASSED  SwapToRamOK
A03500 Network              PASSED  Connected
A03510 IP Address           RECORD  172.16.220.201
A03530 Domain Name          RECORD  NotFound
A03540 /etc/hosts Format    FAILED  No entry found
A03550 DNS Lookup           FAILED  nslookup host.domain
A03600 ip_local_port_range  PASSED  RangeOK
A04305 OL7 Server RPMs OK?  SKIPPED NotOL7
A04315 RHEL7 Server RPMs OK SKIPPED NotRedHat
A04321 SLES12 Server RPMs O SKIPPED NotSuSE
A04325 SLES15 Server RPMs O SKIPPED NotSuSE
Result file: output/collect/DB_HCVE_A_DB19c_lin_res.htm

–rac环境收集rda信息

[oracle@erp51:/home/oracle/rda]$./rda.sh -vXRda start CLOUD -p Rac_Assessment
Creating collection "output" ...
        - Trying to identify the domain ... (can take time)
Enter the network domain name for this server
> 

Warning: No domain has been specified
Do you want to specify a new value (Y/N)? n


 Inside DFT scenario, define the input and profile targets
 Inside DFT scenario, check Oracle home or Middleware presence
        - RDA:DCbegin ...
------------------------------------------------------------------------------
RDA.BEGIN: Initializes the Data Collection
------------------------------------------------------------------------------
Enter the Oracle home to be used for data analysis
Press Return to accept the default (/u01/app/oracle/product/19.0.0/db_1)
> 

        - RDA:DCconfig ...
------------------------------------------------------------------------------
RDA.CONFIG: Collects Key Configuration Information
------------------------------------------------------------------------------
        - RDA:DCocm ...
------------------------------------------------------------------------------
RDA.OCM: Set up the Configuration Manager Interface
------------------------------------------------------------------------------
        - RDA:DCstatus ...
------------------------------------------------------------------------------
RDA.STATUS: Produces the Remote Data Collection Reports
------------------------------------------------------------------------------
        - RDA:DCload ...
------------------------------------------------------------------------------
RDA.LOAD: Produces the External Collection Reports
------------------------------------------------------------------------------
        - RDA:DCfilter ...
------------------------------------------------------------------------------
RDA.FILTER: Controls Report Content Filtering
------------------------------------------------------------------------------
        - RDA:DCend ...
------------------------------------------------------------------------------
RDA.END: Finalizes the Data Collection
------------------------------------------------------------------------------
In CRS start scenario, getting nodes using /u01/app/19.0.0/grid/bin/olsnodes
2>/dev/null
------------------------------------------------------------------------------
Requesting common cluster information
------------------------------------------------------------------------------
Enter the absolute path of the directory where RDA will be installed on the
remote nodes.
Press Return to accept the default (/home/oracle/rda)
> 

Do you want RDA to use an alternative login to execute remote requests (Y/N)?
Press Return to accept the default (N)
> y

Enter the login to execute requests on remote nodes
> 

In CRS start scenario, getting local node information
In CRS start scenario, pinging node erp52
------------------------------------------------------------------------------
Requesting information for node erp51
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node erp51
Press Return to accept the default (/u01/app/oracle/product/19.0.0/db_1)
> 

Enter the Oracle SID to be analyzed on the node erp51
Press Return to accept the default (erpdb1)
> 

------------------------------------------------------------------------------
Requesting information for node erp52
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node erp52
Press Return to accept the default (/u01/app/oracle/product/19.0.0/db_1)
> 

Enter the Oracle SID to be analyzed on the node erp52
> 

        - RDA:DCremote ...
------------------------------------------------------------------------------
RAC Setup Summary
------------------------------------------------------------------------------
Nodes:
. NOD001  erp51/erpdb1
. NOD002  erp52/
2 nodes found

[oracle@erp51:/home/oracle/rda]$./rda.sh 
------------------------------------------------------------------------------
RDA Data Collection Started 21-Oct-2022 02:03:59
------------------------------------------------------------------------------
Processing RDA.BEGIN module ...
Processing RDA.CONFIG module ...
Processing RDA.REMOTE module ...
Processing RDA.END module ...
------------------------------------------------------------------------------
RDA Data Collection Ended 21-Oct-2022 02:15:28
------------------------------------------------------------------------------
  You can review the reports by transferring the /home/oracle/rda/output
  directory structure to a location where you have web-browser access. Then,
  point your browser at this file to display the reports:
    RDA__start.htm

  Based on your server configuration, some possible alternative approaches
  are:
  - If your client computer with a browser has access to a web-shared
    directory, copy the /home/oracle/rda/output directory structure to the
    web-shared directory and visit this URL:
    http://machine:port/web_shared_directory/RDA__start.htm
  or
  - If your client computer with a browser has FTP access to the server
    computer with the /home/oracle/rda/output directory, visit this URL:
    ftp://root@erp51/rda/output

  If this file was generated to assist in resolving a Service Request, send
  /home/oracle/rda/RDA_output_erp51.zip to Oracle Support by uploading the
  file via My Oracle Support. When sending with FTP, make sure you send it in
  BINARY format.

  Note: Do not submit any health, payment card or other sensitive production
  data that requires protections greater than those specified in the Oracle
  GCS Security Practices
  (http://www.oracle.com/us/support/library/customer-support-security-practices-069170.pdf).
  Information on how to remove data from your submission is available at
  https://support.oracle.com/rs?type=doc&id=1227943.1

2.2.2、检查定义的用户限制(ulimit)设置(仅限UNIX)

ulimit -a

2.2.3、验证配置的交换空间

1 GB到2 GB之间:1.5倍于RAM的大小
2 GB到16 GB之间:等于RAM的大小
16 GB以上:16 GB

2.2.4、检查操作系统错误日志和跟踪文件

还要检查问题发生时生成的跟踪文件。查找出现问题时的“平均负载”以及内存和交换状态。

2.2.5、用于监控资源使用情况的操作系统监视器

前面的行动点通常会调查操作系统配置问题，但是该问题也可能是由临时资源高峰或系统过载引起的。请检查问题出现在alert.log中的时间，如果它总是在同一时间发生，那么您正在处理一个临时的资源高峰。这通常发生在系统负载非常高的时候

OS Watcher是一款Oracle工具，允许您从操作系统的角度监控系统

2.2.6、由Linux功能地址空间布局随机化引起的问题(ASLR

# /sbin/sysctl -a | grep randomize
kernel.randomize_va_space = 1

如果该参数设置为0以外的任何值，则使用ASLR.解决办法是禁用ASLR

2.3、检查数据库配置

2.3.1、设置PGA_AGGREGATE_TARGET=TRUE

参数pga_aggregate_target，因此必须设置为一个数字才能正常工作。

2.3.2、验证PRE_PAGE_SGA设置为TRUE

PRE_PAGE_SGA指示Oracle在实例启动时将整个SGA读入活动内存。然后为SGA的每个页面预构建操作系统页表条目。此设置可能会增加实例启动所需的时间，但可能会减少Oracle在启动后达到其全部性能所需的时间。PRE_PAGE_SGA会增加进程的启动持续时间，因为每个启动的进程都必须访问SGA中的每个页面，这可能会导致PMON进程需要更长的时间来启动，并超过默认情况下120秒的超时，从而导致实例启动失败。

将PRE_PAGE_SGA设置为TRUE会增加进程的启动持续时间，因为每个启动的进程都必须访问SGA中的每个页面，但是，如果您的系统频繁地创建和销毁进程(例如，不断地登录和注销),开销会很大。

检查PRE_PAGE_SGAA是否设置为真
-或者-
验证函数:ksmprepage()出现的生成跟踪

解决办法:将PRE_PAGE_SGA设置为FALSE将避免此代码执行，因此页面仅在需要时被触及，而不是在进程开始时触及每一页。这可以避免或最大限度地减少问题的发生，但是根本原因仍然是操作系统资源不足

注意:在12.1.0.2和更高版本中，pre_page_sga的默认值已更改为true。Oracle建议对低于12c的数据库版本禁用pre_page_sga。

三、已知的错误

Unpublished Bug 8336987: TB:SH:ORA-00445: BACKGROUND PROCESS “PZ99” DID NOT START AFTER 120 SECONDS
—> Platform 226 - Linux x86-64
该进程一直停留在“D”状态，即内核中的不可中断状态。这是Linux ext3文件系统的一个已知问题，当有很多I/O活动时，文件系统必须刷新它的日志，并且在此过程中所有其他的I/O都会冻结。如果您愿意尝试，可以将ext3文件系统的挂载选项改为包含-o data=writeback，看看这个问题是否会重现。

Unpublished Bug 7562673: FAILED IN SPAWNING MMON SECONDARY AND FLOODING OF MMON TRACES IN AQ TEST
–> Status 92 - Closed, Not a Bug
警告:检测到OS id=4260，M000的缓慢进程衍生。进程执行花费了超过133秒。
原因:您的oracle可执行文件位于远程计算机上。网络问题会延迟加载可执行映像，从而延迟进程执行。将oracle映像/ oracle主目录复制到实例的本地硬盘。

Unpublished Bug 7377880: STARETL: ORA-00445: BACKGROUND PROCESS “M000” DID NOT START AFTER 120 SECONDS
–> Status 32 - Not a Bug
问题似乎是你在你的机器上有太多的活动；修复方法是关闭一些活动。

Unpublished Bug 9866293: GE+RAC: INSTANCE CRASH BECAUSE SMON DIED WITH ORA-00445
–> Status 92 - Closed, Not a Bug
在负载较重的系统上，进程启动可能非常耗时。系统负载高或网络延迟高且映像位于远程的系统可能会有很长的fork/exec时间。在这种特殊情况下，看起来ps和gdb在转储spawnee的诊断时超时了。这很好地表明了高系统负载/网络延迟。

Unpublished Bug 9543620: SQL EXEC PART:ORA-00445: BACKGROUND PROCESS “P059” DID NOT START AFTER 120 SECONDS
–> Status 92 - Closed, Not a Bug
这看起来只是一个与系统负载相关的问题。获取诊断信息时ps和gdb超时通常意味着，可能存在NFS延迟问题，映像或相关库是远程的，或者如果它们是本地的，则可能存在磁盘延迟问题。

简而言之，这些错误中提到的可能原因有:

1.你的机器活动太多。
2.NFS延迟问题。
3.磁盘延迟问题(影响I/O)。
4.网络延迟。

四、参考文档

https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/server-configuration-checklist-for-oracle-database-installation.html#GUID-CD4657FB-2DDC-4B30-AAB4-2C927045A86D

Troubleshooting Guide (TSG) - Ksvcreate: Process(xxxx) Creation Failed / ORA-00445: Background Process "xxxx" Did Not Start After n Seconds (Doc ID 1379200.1)

Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2) (Doc ID 169706.1)

Remote Diagnostic Agent (RDA) - RAC Cluster Guide (Doc ID 359395.1)

墨力计划

「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」

关注作者