Cloning a standalone 12c grid infrastructure home (aka copying oracle restart)



Cloning using the oracle provided perl utility is seemingly one of the most useful features that oracle provide (and seemingly the most underused - why does no-one use this?).

I've blogged about it before here for the normal database home http://dbaharrison.blogspot.com/2011/02/cloning-oracle-software-why-does-nobody.html

We're currently beginning an ASM rollout across the estate (all single instance - no RAC here) and I don't want to do loads of GUI installs - it's a pain and it involves co-ordination with our service provider to run various commands as root during that gui install - which is easier said than done. I'd rather just get the software on there, get the root commands run as and wen by the supplier and then come back and finish the asm/listener setup later.

So is this possible to do with cloning - thats what i wanted to find out, the docs imply it will work but i couldn;t find a specific example for s standalone instance, and there was no exact post anywhere i found on the net.

So i decided to just try it out and see if it would work, in short it does work and it is pretty simple but it took me a fair while to figure out all the steps and get the syntax right - the following is the abridged version of that journey (when you get to the end you'll be asking was that abridged? - well yes it was is the answer to that)

So here goes:

First up we tar up an existing GI install that we know works ok (it doesn't need to be stopped)

tar -cvf /tmp/grid.tar $GI_HOME


We then copy that tar file to the destination server using whatever tool you see fit scp/sftp etc. We then untar in my case (you could have used zip earlier) that file into the destination directory - i used the same path as the original server but as with the database software this shouldn't matter

tar -xvf /oracle/product/12.1.0/grid.tar

So now we have a home that appears at first glance to be 'valid', however it's not it needs to be registered in the inventory and recompiled - this is where clone.pl comes in - lets now run that - we need to specify 3 parameters when we call it, ORACLE_HOME (self explanatory), ORACLE_BASE (ditto) and ORACLE_HOME_NAME (this is just the 'label' name in the inventory

oracle@server-name:/oracle/product/12.1.0/grid/clone/bin> ./clone.pl ORACLE_HOME=/oracle/product/12.1.0/grid ORACLE_HOME_NAME=GI12c ORACLE_BASE=/oracle/product/12.1.0


oracle@server-name:/oracle/product/12.1.0/grid/clone/bin> ./clone.pl ORACLE_HOME=/oracle/product/12.1.0/grid ORACLE_HOME_NAME=GI12c ORACLE_BASE=/oracle/product/12.1.0
./runInstaller -clone -waitForCompletion  "ORACLE_HOME=/oracle/product/12.1.0/grid" "ORACLE_HOME_NAME=GI12c" "ORACLE_BASE=/oracle/product/12.1.0" -silent -paramFile

/oracle/product/12.1.0/grid/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 1161 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 100295 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-05-13_03-52-49PM. Please wait ...You can find the log of this install session at:
 /oracle/oraInventory/logs/cloneActions2014-05-13_03-52-49PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..................................................   90% Done.
..................................................   95% Done.

Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of GI12c was successful.
Please check '/oracle/oraInventory/logs/cloneActions2014-05-13_03-52-49PM.log' for more details.

As a root user, execute the following script(s):
        1. /oracle/product/12.1.0/grid/root.sh


..................................................   100% Done.
oracle@server-name:/oracle/product/12.1.0/grid/clone/bin>


And bang clone done (in less than a minute for me), now we just need to get our friendly unix team to run root.sh shown above along with the standard script to setup GI for the first time - shown below here

    /oracle/product/12.1.0/grid/perl/bin/perl -I /oracle/product/12.1.0/grid/perl/lib -I /oracle/product/12.1.0/grid/crs/install  /oracle/product/12.1.0/grid/crs/install/roothas.pl


they ran them for me (eventually) and sent me the output



server-name:~ # id
uid=0(root) gid=0(root) groups=0(root)
server-name:~ # /oracle/product/12.1.0/grid/root.sh
Check /oracle/product/12.1.0/grid/install/root_server-name_2014-05-14_01-23-47.log for the output of root script
server-name:~ # /oracle/product/12.1.0/grid/perl/bin/perl -I /oracle/product/12.1.0/grid/perl/lib -I /oracle/product/12.1.0/grid/crs/install  /oracle/product/12.1.0/grid/crs/install/roothas.pl
Using configuration parameter file: /oracle/product/12.1.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node server-name successfully pinned.
2014/05/14 01:24:08 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab'


server 2014/05/14 01:24:35     /oracle/product/12.1.0/grid/cdata/server-name/backup_20140514_012435.olr
2014/05/14 01:24:41 CLSRSC-327: Successfully configured Oracle Grid Infrastructure for a Standalone Server

So looking good, now we just need to run the following script as the GI owner

    /oracle/product/12.1.0/grid/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/12.1.0/grid -defaultHomeName CLUSTER_NODES= CRS=TRUE

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 100287 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.


So thats it the software at least is cloned and should be working, easy huh? The next bit took me a little more time to sort out.......

Lets try and run the asmca (again using command line version not GUI here)

 I specify everything that should be needed to do this in one pass

  asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx -diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL

And we get this output

An ASM Instance with SID +ASM is already detected. ASM failed to start. Check /oracle/product/12.1.0/cfgtoollogs/asmca/asmca-140514AM073949.log for details.

 After a quick investigation this was down to the entry i added in the oratab to make setting the environment easy for me - lets comment out +ASM from oratab file as it doesn't exist yet and it's confusing the asmca tool.

Lets try it again

Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER_+ASM'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.asm' on 'server-name' failed


OK - still an obvious message, a listener needs to be there for it to register against and there isn't one (at least not the one in wants anyway)

 Lets add a default one on 1521

[oracle@server-name]:GI:[~]# srvctl add listener

[oracle@server-name]:GI:[~]# srvctl start listener
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5016: Process "/oracle/product/12.1.0/grid/bin/lsnrctl" spawned by agent "/oracle/product/12.1.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in

"/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-5016: Process "/oracle/product/12.1.0/grid/bin/lsnrctl" spawned by agent "/oracle/product/12.1.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in

"/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'server-name' failed


 Again after investigating the problem was the listener.ora from the cloned home - it contained the old server name in it - causing the issue, lets remove listener.ora from GI home

 srvctl start listener

now starts OK (well no errors anyway). Lets check the status of it.

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle    5591     1  0 08:21 ?        00:00:02 /oracle/product/11.2.0.2.0/bin/tnslsnr LISTENER -inherit
oracle   28305     1  0 14:44 ?        00:00:00 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle   32420 24977  0 14:45 pts/0    00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


Hmmm, 2 listeners with the same name that shouldn;t be possible should it..... The lsnrctl status command shows the status of the wrong listener. This isn;t going to work out lets get rid of the listener i tried to add and instead just have one listener for everything on the box

Lets stop the GI one and hope it does just stop the GI one.....

srvctl stop listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle    5591     1  0 08:21 ?        00:00:02 /oracle/product/11.2.0.2.0/bin/tnslsnr LISTENER -inherit
oracle   15980 24977  0 14:49 pts/0    00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


So it has stopped the correct one ......

 Lets point GI at the already existing listener.ora in /etc and use that as the single listener
 so we need to add it again and then modify the config. So first up i stop the 11.2.0.2 listener and then start it manually using lsnrctl from the GI home. I then add the listener to restart as previously with srvctl add listener

I then need to amend the environment to tell it to use the listener.ora from/etc

 [oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl setenv listener -t TNS_ADMIN=/etc
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
TNS_ADMIN=/etc
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


So now that's done, i should be able to 'start' it (though it's already running so it's just a check essentially)

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle    5591     1  0 08:21 ?        00:00:02 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -inherit
oracle   15293 24977  0 14:57 pts/0    00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


So that all seems ok, lets do a clean stop and start to confirm everything is OK, the stop works fine but when i start it again this happens:


[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5010: Update of configuration file "/etc/listener.ora" failed: details at "(:CLSN00014:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'server-name' failed


At first i tried messing around with permissions on the file in /etc but it just wasn't fixing it. So i decided to switch off from using /etc and just use the file from the 'normal' $ORACLE_HOME location - so first we switch off TNS_ADMIN then try again

srvctl setenv listener -t TNS_ADMIN=
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle   24526     1  0 15:07 ?        00:00:00 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle   26436 24977  0 15:08 pts/0    00:00:00 grep lsnr



this has now worked ok, switching the location reveals the problem, when the process starts it wants to create a backup file - as /etc is owned by root it can't do it there...... (see example backup file below)


[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ls -lrt
total 24

-rw-r--r-- 1 oracle oinstall 3083 May 14 15:06 listener.ora.bak.server-name
-rw-r--r-- 1 oracle oinstall 3221 May 14 15:07 listener.ora


Interestingly though the backup is taken in $ORACLE_HOME but /etc is actually overriding the $ORACLE_HOME anyway - see here.......look at the listener parameter file location....... so at some point we need to get the /etc file removed it will confuse things at some point in the future.

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# lsnrctl stat

LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 14-MAY-2014 15:13:55

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server-name.e-ssi.net)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.1.0 - Production
Start Date                14-MAY-2014 15:07:41
Uptime                    0 days 0 hr. 6 min. 13 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /etc/listener.ora
Listener Log File         /oracle/product/12.1.0/diag/tnslsnr/server-name/listener/alert/log.xml

Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1531)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1527)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1524)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1525)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1526)))



Now the listener on this server was actually listening on loads of ports - but none of them was 1521
however the output of this

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1521
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#



shows 1521 - so what is the point of this config?

lsof confirms 1521 not in use

Now at this point i got very confused about this whole listener endpoints thing  and the mismatch between the config above and what is actually in the running listener. I ignored that for the moment and blundered on regardless

So now we have a listener running in oracle restart with the correct config to service all of my running databases - now we can use that for ASM. I need to add listener to tnsnames.ora so asmca can find it and i need to tell asmca what its called ( i did at one point try to name it with the default name it seems to want but that caused more pain so i just stuck with listener)

Here we go again then - note the extra parameter at the end

  asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER

and......

Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.asm' on 'server-name' failed






So it can't fine the name listener in the tnsnames file - the one i put there 2 mins ago...

At this point i realised /etc was likely involved again.....

Sure enough the file exists but is empty


[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# cat /etc/tnsnames.ora
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ls -l /etc/tnsnames.ora
-rw-r--r-- 1 oracle oinstall 0 Dec 19 08:16 /etc/tnsnames.ora



Setting TNS_ADMIN and then i can tnsping it ok

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# export TNS_ADMIN=/oracle/product/12.1.0/grid/network/admin
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#  tnsping listener

TNS Ping Utility for Linux: Version 12.1.0.1.0 - Production on 14-MAY-2014 20:45:55

Copyright (c) 1997, 2013, Oracle.  All rights reserved.

Used parameter files:
/oracle/product/12.1.0/grid/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)

(PORT = 1531))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1527))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1524))) (DESCRIPTION

= (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1525))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1526))))
OK (0 msec)
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


So lets set TNS_ADMIN in the environment and try again

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#  export TNS_ADMIN=/oracle/product/12.1.0/grid/network/admin

  asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER

Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.asm' on 'server-name' failed


hmm - that didn;t work - lets just put the tnsnames entry in /etc where it wants to find it


[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# unset TNS_ADMIN

  asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER

ASM created and started successfully.

Disk Group DATA created successfully.

Disk Group FRA created successfully.


Wow - it only worked!

lets see what state everything is in

crsctl stat res -t

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE server-name                  Not All Endpoints Re
                                                             gistered,STABLE
ora.asm
               ONLINE  ONLINE       server-name                  Started,STABLE
ora.ons
               OFFLINE OFFLINE      server-name                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server-name                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       server-name                  STABLE
--------------------------------------------------------------------------------


So ASM looks good, but whats going on with the listener (should have checked this earlier) - this must be due to the mismatch between the restart config and the running listener

[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1521

lsnrctl stat

Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1531)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1527)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1524)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1525)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1526)))

So how do we fix - we can use srvctl to update the ports it should be checking the listener is actually on

 srvctl modify listener -p "TCP:1523,1531,1527,1524,1525,1526"
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#  srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1523,1531,1527,1524,1525,1526
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#


 So thats now updated OK - lets check the status


[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details                                                                                  
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE server-name                  Not All Endpoints R                                                                             e
                                                             gistered,STABLE
ora.asm
               ONLINE  ONLINE       server-name                  Started,STABLE
ora.ons
               OFFLINE OFFLINE      server-name                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server-name                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       server-name                  STABLE
--------------------------------------------------------------------------------




Still not right - lets restart the listener and see what happens

[oracle@server-name]:GI:[~]# srvctl stop listener
[oracle@server-name]:GI:[~]# srvctl start listener
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server-name                  STABLE
ora.asm
               ONLINE  ONLINE       server-name                  Started,STABLE
ora.ons
               OFFLINE OFFLINE      server-name                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server-name                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       server-name                  STABLE
--------------------------------------------------------------------------------
[oracle@server-name]:GI:[~]#


Brilliant - all now looks ok - lets restart the whole stack and check it comes back ok

[oracle@server-name]:GI:[~]# srvctl stop asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified
[oracle@server-name]:GI:[~]# srvctl stop asm -f


[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      server-name                  STABLE
ora.FRA.dg
               ONLINE  OFFLINE      server-name                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server-name                  STABLE
ora.asm
               OFFLINE OFFLINE      server-name                  Instance Shutdown,ST
                                                             ABLE
ora.ons
               OFFLINE OFFLINE      server-name                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server-name                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       server-name                  STABLE
--------------------------------------------------------------------------------



[oracle@server-name]:GI:[~]# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'server-name'
CRS-2673: Attempting to stop 'ora.evmd' on 'server-name'
CRS-2677: Stop of 'ora.evmd' on 'server-name' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'server-name'
CRS-2677: Stop of 'ora.cssd' on 'server-name' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'server-name' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[oracle@server-name]:GI:[~]# crsctl start has

CRS-4123: Oracle High Availability Services has been started.
[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]#  crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       server-name                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server-name                  STABLE
ora.asm
               ONLINE  ONLINE       server-name                  Started,STABLE
ora.ons
               OFFLINE OFFLINE      server-name                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server-name                  STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  INTERMEDIATE server-name                  STABLE
--------------------------------------------------------------------------------


So everything but evmd looks ok - lets wait a little longer and afew secs later we get this

ora.evmd
      1        ONLINE  ONLINE       server-name                  STABLE
--------------------------------------------------------------------------------



so all seems well - i successfully create files in ASM from a database instance on the server
We cloned oracle restart! back slapping all round - job well done and all that.......


And i even sorted out my earlier confusion about the whole endpoints config.



Note to self - sort out /etc......



7 comments:

  1. Thanks for post , did You copied GI binaries when clusterware stack was up ? AFAIK it is not supported and You have to unlock GI and stop clusterware before .
    Regards
    GG

    ReplyDelete
  2. Hi GG,
    Everything was done online....
    And everything seems fine.....

    I guess problems could possible be revealed when we try to patch this...

    I'll post again if this all goes horribly wrong :-)

    Cheers,
    Rich

    ReplyDelete
  3. Thank you for posting CRS-5010 resolution. Helped me over last hurdle with configuring listener on 2nd network (12c feature). I now have clusterware on 10.x.x.x and 2nd listener on 192.x.x.x for testing.

    ReplyDelete
  4. Nice post thanks

    ReplyDelete
  5. Nice post very useful! thank you

    ReplyDelete