CRS-4013: This command is not supported in a single-node configuration

Issue: Getting CRS-4013 , CRS-4000 error while starting up cluster services



CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.

One of the cluster node is reporting the CRS-4013 , CRS-4000 error while other node is up in 2 node RAC cluster 

Solution:

1. Error while starting up cluster on problematic node(mytestlab1283)

[ora12c@mytestlab1283 ~]$ ps -ef | grep pmon
ora12c  8572 8540  0 10:59 pts/0    00:00:00 grep --color=auto pmon

[ora12c@mytestlab1283 ~]$ . ./.bash_profile_asm
[ora12c@mytestlab1283 ~]$ which crsctl
/rdbms/app/12.1.0.2/grid/bin/crsctl

[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.

2.  Check resource status on working node (mytestlab12834) 

[ora12c@mytestlab1284 ~]$ ps -ef | grep pmon
ora12c    2601  2550  0 11:09 pts/0  00:00:00 grep --color=auto pmon
ora12c   23415     1  0 May09 ?      00:10:50 asm_pmon_+ASM2
ora12c   23517     1  0 May09 ?      00:12:00 ora_pmon_oemdbmn2

[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl stop crs -f
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.

[ora12c@mytestlab1283 ~]$ crsctl check css
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.

[ora12c@mytestlab1284 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[ora12c@mytestlab1284 ~]$ crsctl status res -t
------------------------------------------------------------
Name    Target State   Server   State details Local Resources
-------------------------------------------------------------
ora.DATA.dg ONLINE  ONLINE       mytestlab1284            STABLE
ora.FRA.dg  ONLINE  ONLINE       mytestlab1284            STABLE
ora.LISTENER.lsnr ONLINE  ONLINE   mytestlab1284          STABLE
ora.OCR_VOTING.dg  ONLINE  ONLINE   mytestlab1284         STABLE
ora.asm  ONLINE  ONLINE       mytestlab1284         Started,STABLE
ora.net1.network   ONLINE  ONLINE   mytestlab1284         STABLE
ora.ons   ONLINE  ONLINE    mytestlab1284                 STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr 1  ONLINE  ONLINE mytestlab1284    STABLE
ora.LISTENER_SCAN2.lsnr 1  ONLINE  ONLINE  mytestlab1284   STABLE
ora.LISTENER_SCAN3.lsnr 1 ONLINE  ONLINE   mytestlab1284   STABLE
ora.MGMTLSNR      1       OFFLINE  OFFLINE                 STABLE 
ora.cvu       1    ONLINE ONLINE    mytestlab1284          STABLE
ora.mytestlab1283.vip 1 ONLINE INTERMEDIATE mytestlab1284  FAILED OVER,STABLE
ora.mytestlab1284.vip  1 ONLINE ONLINE mytestlab1284       STABLE
ora.oc4j    1    ONLINE ONLINE    mytestlab1284            STABLE
ora.oemdbmn.db  1    ONLINE  OFFLINE                       STABLE
                2    ONLINE  ONLINE  mytestlab1284 Open,STABLE 
ora.scan1.vip    1     ONLINE  ONLINE   mytestlab1284    STABLE
ora.scan2.vip    1     ONLINE  ONLINE   mytestlab1284    STABLE
ora.scan3.vip    1     ONLINE  ONLINE   mytestlab1284    STABLE
-------------------------------------------------------------------


[ora12c@mytestlab1284 trace]$ srvctl config database -d oemdbmn
Database unique name: oemdbmn
Database name: oemdbmn
Oracle home: /rdbms/app/oracle/12.1.0.2/db
Oracle user: ora12c
Spfile: +DATA/OEMDBMN/PARAMETERFILE/spfile.268.943049417
Password file: +DATA/OEMDBMN/PASSWORD/pwdoemdbmn.256.943049111
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: oemdbmn1,oemdbmn2
Configured nodes: mytestlab1283,mytestlab1284
Database is administrator managed


3.  Stop crs on working node (mytestlab12834) 

[ora12c@mytestlab1284 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl stop crs
[sudo] password for ora12c:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.crsd' on 'mytestlab1284'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.oemdbmn.db' on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.oc4j' on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'mytestlab1284'
.
.
.
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'mytestlab1284' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'mytestlab1284'
CRS-2677: Stop of 'ora.cssd' on 'mytestlab1284' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'mytestlab1284'
CRS-2677: Stop of 'ora.gipcd' on 'mytestlab1284' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mytestlab1284' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[ora12c@mytestlab1284 ~]$ ps -ef | grep pmon
ora12c    6311  5970  0 11:29 pts/1    00:00:00 grep --color=auto pmon

4. Execute below on problematic node (mytestlab1283) 

[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.


sudo /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.sh -deconfig -force

sudo /rdbms/app/12.1.0.2/grid/root.sh




5. Verify logs and crs process status on OS 

[ora12c@mytestlab1283 ~]$ ps -ef | grep -i crs
root     25735 25646  0 12:55 pts/0    00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/config/rootconfig.sh
root     25744 25735  0 12:55 pts/0    00:00:00 /rdbms/app/12.1.0.2/grid/perl/bin/perl -I/rdbms/app/12.1.0.2/grid/perl/lib -I/rdbms/app/12.1.0.2/grid/crs/install /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.pl
root     25763 25744  0 12:55 pts/0    00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /rdbms/app/12.1.0.2/grid
ora12c   26600 26372  0 12:57 pts/1    00:00:00 grep --color=auto -i crs

[ora12c@mytestlab1283 ~]$ ps -ef | grep -i crs
root     25735 25646  0 12:55 pts/0    00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/config/rootconfig.sh
root     25744 25735  0 12:55 pts/0    00:00:00 /rdbms/app/12.1.0.2/grid/perl/bin/perl -I/rdbms/app/12.1.0.2/grid/perl/lib -I/rdbms/app/12.1.0.2/grid/crs/install /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.pl
root     28153 25744  0 12:58 pts/0    00:00:00 sh -c /bin/su ora12c -c ' echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES ' 2>&1
root     28154 28153  0 12:58 pts/0    00:00:00 /bin/su ora12c -c  echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c   28155 28154  0 12:58 ?        00:00:00 bash -c  echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROO CRS_INITRES
ora12c   28156 28155  0 12:58 ?        00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c   28159 28156  0 12:58 ?        00:00:00 /rdbms/app/12.1.0.2/grid/jdk/bin/java -DTRACING.ENABLED=true -DTRACING.LEVEL=2 -classpath /rdbms/app/12.1.0.2/grid/jlib/srvm.jar:/rdbms/app/12.1.0.2/grid/jlib/srvmhas.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/OraCheckPoint.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/OraInstaller.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/xmlparserv2.jar oracle.ops.util.ClusterUtil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c   28172 26372  0 12:58 pts/1    00:00:00 grep --color=auto -i crs


[ora12c@mytestlab1283 ~]$ tail -f /rdbms/app/12.1.0.2/grid/install/root_mytestlab1283.vsnl.co.in_2017-08-10_13-01-13.log
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /rdbms/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/08/10 13:01:14 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/08/10 13:03:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/08/10 13:03:01 CLSRSC-363: User ignored prerequisites during installation
2017/08/10 13:04:06 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.gipcd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.evmd' on 'mytestlab1283'
CRS-2677: Stop of 'ora.gpnpd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.evmd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'mytestlab1283' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mytestlab1283' has completed
.
.
.
.
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'mytestlab1283'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'mytestlab1283' succeeded
CRS-2672: Attempting to start 'ora.oemdbmn.db' on 'mytestlab1283'
CRS-2676: Start of 'ora.oc4j' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.oemdbmn.db' on 'mytestlab1283' succeeded
CRS-6016: Resource auto-start has completed for server mytestlab1283
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/08/10 13:07:53 CLSRSC-343: Successfully started Oracle Clusterware stack
PRKO-2190 : VIP exists for node mytestlab1283, VIP name mytestlab1283-vip.vsnl.co.in
PRKZ-1072 : SCAN name "XPDBoemdbmn" is already registered on network 1
PRCS-1028 : Single Client Access Name (SCAN) listener resources already exist on network 1
PRCN-3004 : Listener MGMTLSNR already exists
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'mytestlab1283'
.
.
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'mytestlab1283'
Preparing packages...
cvuqdisk-1.0.9-1.x86_64
PRCN-3004 : Listener LISTENER already exists
PRCC-1015 : LISTENER was already running on mytestlab1283
PRCR-1004 : Resource ora.LISTENER.lsnr is already running

2017/08/10 13:08:36 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

No comments:

Post a Comment

My Popular Posts