APPLIES TO:

Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Cloud Schema Service - Version N/A and later
Information in this document applies to any platform. *** ***

PURPOSE

The Purpose of this Document is to troubleshoot the generic Error

Heartbeat failed to connect to standby

in the Primary ALERT.LOG. It shows most common and possible Causes along with Solutions to get rid of this Problem.

The Errors in general look like this:

PING[ARCn]: Heartbeat failed to connect to standby ‘’. Error is xxxxx

ORA-16191/ORA-1017

ORA-3135 :The Communication between the ARCn and the RFS-Process died unexpectedly

ORA-12514 : The Standby Database is down or the specified Service you want to connect to is not registered with Listener on the Standby Site

Many more Common errors described in this document.

TROUBLESHOOTING STEPS

Introduction

Once Log Transport Services from a Primary to a Standby Database are setup correctly and the Archive Destination is enabled and active, there will be a Heartbeat-Ping between the Primary and Standby Database. This Ping is being performed by a dedicated ARCn-Process on the Primary Database to an associated RFS-Process on the Standby. You can find out about this Process if you have a Look into the ALERT.LOG of the Primary Database and search for an Entry like this:

ARC1 started with pid=21, OS id=6585
…
ARC1: Becoming the heartbeat ARCH
-> In this Case we can find that ARC1 with pid 21 (OS-pid 6585) is the current Heartbeat ARCn-Process.

This Process tries to ping an associated RFS-Process on the Standby Database. If you look into the Standby ALERT.LOG you can find Entries like this:

RFS[2]: Assigned to RFS process 6621
RFS[2]: Identified database type as ‘physical standby’: Client is ARCH pid 6585
-> So here RFS-Process with OS-pid 6621 is the corresponding RFS-Process for the Heartbeat Ping

Note that this can be quite dynamic since RFS-Processes are created and terminated on Need.
If the Primary ARCn-Process is not able to reach a corresponding RFS-Process this Error is raised in the Primary ALERT.LOG together with the corresponding Reason.

Diagnosing Problems:

If the Heartbeat Ping fails you typically get an Error along with this Message in the following Section we discuss common Errors and how to solve them.

The Errors in general look like this:

PING[ARCn]: Heartbeat failed to connect to standby ‘’. Error is xxxxx

General Points to verify first:

• The Standby Database must at least be mounted to create a RFS-Process and make the Heartbeat Ping happen. So first of all you should always verify if the Standby Database is at least in ‘mount’-Status and registered its Services with the correct Listener on the Standby Site.
• Verify if you are able to connect to the Standby Database from the Primary Site using the same Connect Descriptor or TNS-Alias used in the corresponding log_archive_dest_n:

SQL> connect sys/@ as sysdba

-> if this succeeds then it’s a Problem specific on the Database where if you get the same Error here there is a general (mostly Setup or Configuration) Problem

• ARCn- and Database Processes read the TNSNAMES.ORA only when they are started or Log Transport Services are restarted. So if you make Modifications in the TNSNAMES.ORA on the Alias for the Standby Database the ARCn-Processes do not get aware of this Change unless they are restarted or you stop and restart Log Transport Services, ie. set the corresponding log_archive_dest_state_n to ‘defer’ and back to ‘enable’.
• As an Alternative you can also try to directly put the Connect Identifier into the log_archive_dest_n which will avoid having to restart the ARCn-Processes or taking care of the correct TNSNAMES.ORA to be used

Here are common Errors and Solutions:

• ORA-12514

The Standby Database is down or the specified Service you want to connect to is not registered with Listener on the Standby Site.
- Verify the Standby Database is at least in mount-Status
- Ensure the Service used to connect by Log Transport Services is registered with the correct Listener
- Review the TNSNAMES.ORA and ensure the Connection Details (Host, Protocol and Port) are correct

• ORA-12541

The Log Transport Services cannot detect a Listener on the Destination
- Ensure the Listener is running
- Review the TNSNAMES.ORA and ensure the Connection Details (Host, Protocol and Port) are correct
- Verify in /etc/hosts-File the IP-Address given for the local Node where the Listener is running is defined with it’s real IP-Address rather than the localhost Address (12x.0.x.x) and it matches with the Listener IP-Address or Hostname Resolution

• ORA-12154

The given Connect Descriptor used for Log Transport Services cannot be resolved
- Ensure the TNS-Alias setup with log_archive_dest_n exists in the TNSNAMES.ORA and is valid (Spelling, Brackets,…)
- Try to manually connect to the Standby Database using the same TNS-Alias
- Verify if you modified the TNSNAMES.ORA since the Database ARCn and LNS-Processes started; they may not be aware of the Change. So you may have to kill those so that they get respawned again

• ORA-3135

The Communication between the ARCn and the RFS-Process died unexpectedly.
- Typical Cause are active Firewalls or Routers in the Network Connection between the Primary and Standby Database. Ensure there are no Features enabled on this Equipment which are able to modify TCP-Packets
- The Network is overloaded; ensure there is always sufficient Bandwith available to cope with the current Redo Generation Rate - also see
How To Calculate The Required Network Bandwidth Transfer Of Archivelogs In Dataguard Environments (Note 736755.1)
for Calculation

• ORA-16191/ORA-1017

The Log Transport Services cannot authenticate on the Standby Database
- Ensure Passwordfile has been transfered to the Standby Site correctly
- REMOTE_LOGIN_PASSWORDFILE is setup correct
- Review
Troubleshooting ORA-16191 and ORA-1017 in Data Guard Log Transport Services (Note 1368170.1)
for further Troubleshooting

Workaround :

1. If DG broker is enabled,

alter system set dg_broker_start=false;
alter system set dg_broker_start=true;

2. restart the arc processes on primary/standby side, each node:

ps -ef | grep arc , identify the arc process for your instance
kill -9 , all the arch processes available for this instance will be killed and they will be re-spawned by Oracle.

If database is >=12.1 RDBMS additional processes need to be restarted:
ps -ef | grep tt , identify the arc process for your instance
kill -9 , all the tt processes available for this instance will be killed and they will be re-spawned by Oracle.

3. If issue persists, Verify all the above details. If SQLPLUS connect is working and issue persists on REMOTE log shipping (v$archive_dest) then hard code the tnsnames.ora connect string.

ALTER SYSTEM SET log_archive_dest_2=‘service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=<host/IP address of Standby>)(PORT=)))(CONNECT_DATA=(SERVICE_NAME=)(SERVER=dedicated)))"’,’ lgwr async db_unique_name="" valid_for=(online_logfile,primary_role)’ SCOPE=BOTH;

4. Having hard coded value is a workaround. resting primary database will resolve the issue. This is because after restart primary database will look into the correct TNS_ADMIN.