RAC Troubleshooting

RAC Troubleshooting

This is the one section what will be updated frequently as my experience with RAC grows, as RAC has been around for a while most problems can be resolve with a simple google lookup, but a basic understanding on where to look for the problem is required. In this section I will point you where to look for problems, every instance in the cluster has its own alert logs, which is where you would start to look. Alert logs contain startup and shutdown information, nodes joining and leaving the cluster, etc.

Here is my complete alert log file of my two node RAC starting up.

The cluster itself has a number of log files that can be examined to gain any insight of occurring problems, the table below describes the information that you may need of the CRS components

1. $ORA_CRS_HOME/crs/log : contains trace files for the CRS resources

2. $ORA_CRS_HOME/crs/init : contains trace files for the CRS daemon during startup, a good place to start

3. $ORA_CRS_HOME/css/log : contains cluster reconfigurations, missed check-ins, connects and disconnects from the client CSS listener. Look here to obtain when reboots occur

4. $ORA_CRS_HOME/css/init : contains core dumps from the cluster synchronization service daemon (OCSd)


5. $ORA_CRS_HOME/evm/log : log files for the event volume manager and eventlogger daemon

6. $ORA_CRS_HOME/evm/init : pid and lock files for EVM

7. $ORA_CRS_HOME/srvm/log : log files for Oracle Cluster Registry (OCR)


8. $ORA_CRS_HOME/log : log files for Oracle clusterware which contains diagnostic messages at the Oracle cluster level

As in a normal Oracle single instance environment, a RAC environment contains the standard RDBMS log files, these files are located by the parameter background_dest_dump. The most important of these are

9. $ORACLE_BASE/admin/udump : contains any trace file generated by a user process

10. $ORACLE_BASE/admin/cdump : contains core files that are generated due to a core dump in a user process

Now lets look at a two node startup and the sequence of events

First you must check that the RAC environment is using the connect interconnect, this can be done by either of the following

Logfile ## The location of my alert log, yours may be different

/u01/app/oracle/admin/racdb/bdump/alert_racdb1.log

ifcfg command oifcfg getif

table check select inst_id, pub_ksxpia, picked_ksxpia, ip_ksxpia from x$ksxpia;

Oradebug SQL> oradebug setmypid

SQL> oradebug ipc



Note: check the trace file which can be located by the parameter user_dump_dest

system parameter cluster_interconnects

Note: used to specify which address to use