ORA-00603 ORA-27504: IPC error creating OSD context ORA-27300 ORA-27301 ORA-27302

APPLIES TO:

SYMPTOMS

"ORA-27301: OS failure message: No buffer space available" occurs on DBCS instance with RAC configuration: 

- alert_xxxxxx1.log

2018-04-08T19:54:27.605773+09:00
Errors in file /u01/app/oracle/diag/rdbms/xxxxxx/xxxxxx1/trace/xxxxxx1_j000_4624.trc (incident=136769) (PDBNAME=XXXXXXXXXX):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
XXXXXXXXXX(6):Incident details in: /u01/app/oracle/diag/rdbms/xxxxxx/xxxxxx1/incident/incdir_136769/xxxxxx1_j000_4624_i136769.trc
...
2018-04-08T19:54:42.355736+09:00
Errors in file /u01/app/oracle/diag/rdbms/xxxxxx/xxxxxx1/trace/xxxxxx1_j000_4624.trc (incident=136770) (PDBNAME=XXXXXXXXXX):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
XXXXXXXXXX(6):Incident details in: /u01/app/oracle/diag/rdbms/xxxxxx/xxxxxx1/incident/incdir_136770/xxxxxx1_j000_4624_i136770.trc

CHANGES

CAUSE

Similar problems have been reported also on on-premises RAC database, but as for the case occurs on OPC RAC instances it was investigated on unpublished Bug 26485917.
As a result it was proved that the value of kernel parameter vm.min_free_kbytes was insufficient, and due to that there were cases that No buffer space available occurred from OS side.

vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256 256 32
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 15560 <--------------------
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 4096
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0

As above 15560 was insufficient and it was concluded that vm.min_free_kbytes should be set to 524288.
 

SOLUTION

This problem has been fixed on 18.2.1.

If the current value is not 524288 but 15560 please follow the steps and modify the value manually.

1. Shutdown the database and clusterware.
   
   As oracle user:
   

   [oracle@XXXXXXX ~]$ srvctl stop database -db xxxx

   As root user:
   

   [root@XXXXXXX ~]# crsctl stop crs

2. It is recommended to keep a backup of current sysctl.conf on each node.
   
   
3. Follow below steps and update the value of vm.min_free_kbytes on each node:
   
   a) Check the current value
   As root user:
   

   [root@XXXXXXX ~]# /sbin/sysctl -a | grep min_free_kbytes

   
   b) Open sysctl.conf with a editor like vi and add following. After that, save the file.
   

   vm.min_free_kbytes = 524288

   
   c) Run following command
   As root user:
   

   [root@XXXXXXX ~]# /sbin/sysctl -p

   
   d) Check the current value and confirm the value has been updated.
   As root user:
   

   [root@XXXXXXX ~]# /sbin/sysctl -a | grep min_free_kbytes

4. Reboot the nodes one at a time.
   
   
5. Validate all cluster and database resources is up
   As grid or root user:
   

   [grid@XXXXXXX ~]$ crsctl stat res -t