Switchover failure due to ASM & redo log files.

[Anna Macnaughtan]

Hi Gustav

A new release of Dbvisit Standby with the fix will be available in the next day or two, I will let you know when it is released.
Regards Anna

[Anna Macnaughtan]

Hi Gustav

You can download the new release 6.0.18 from the Dbvisit website. We would appreciate if you could test it and provide us with your feedback.

Regards Anna

[Gustav]

Hi Anna,
Impressively fast!
We'll do that today, I'll let you know how it goes as soon as we've tried it.

[Gustav]

Problems encountered during/after update:


The dbvserverd processes didn't shut down properly as part of the automated install (though that's what the User's Guide says the program will do). Easily taken care of manually (using dbserverd stop as seen in the User's Guide)


When trying to run dbvisit -i VISITD on the primary (and standby!), the result is this:

Originally Posted by LINUX

[oracle@dbvisit1 dbvisit]$ dbvisit -i VISITD
================================================== ===========
Dbvisit Standby Database Technology
Variable ORACLE_SID_ASM_DEST is declared twice
under different categories
in Dbvisit Database configuration (DDC) file:
/usr/local/dbvisit/standby/dbv_VISITD.env
Dbvisit Standby cannot continue.
Please edit /usr/local/dbvisit/standby/dbv_VISITD.env and remove duplicates.
Exiting.
================================================== ===========

#-out the second time the variable was set in the file (not quite set, since the variable was empty there)
Tried again, got:

Originally Posted by LINUX

[oracle@dbvisit1 dbvisit]$ dbvisit -i VISITD
================================================== ===========
Dbvisit Standby Database Technology
Variable ORACLE_SID_ASM is declared twice
under different categories
in Dbvisit Database configuration (DDC) file:
/usr/local/dbvisit/standby/dbv_VISITD.env
Dbvisit Standby cannot continue.
Please edit /usr/local/dbvisit/standby/dbv_VISITD.env and remove duplicates.
Exiting.
================================================== ===========

Same solution. Seems to be working....

Ran into this issue:

Originally Posted by LINUX

[oracle@dbvisit1 etc]$ dbvisit VISITD
================================================== ===========
Dbvisit Standby Database Technology (6.0.18.8316) (pid 31073)
dbvisit started on dbvisit1: Mon Feb 13 10:15:32 2012 (CLI)
================================================== ===========

Obtaining information from standby database (RUN_INSPECT=Y)...
201202131015 - Dbvisit Standby is incompatible with current installed
repository. Current Dbvisit Standby release is: 6.0 and repository is 5.8.
Please upgrade the repository before proceeding.

In the web interface please go to Home > Setup > Update > Manage Repository.

For command line interface please choose option 5).
Dbvisit Standby terminated.
Return code = 601

(Tracefile required if contacting Dbvisit Standby support:
/usr/tmp/31073_dbvisit_VISITD_201202131015.trc (server:dbvisit1))

This was complicated by the default port of dbvserver having switched from 8081 to 8443 and running https now. Not hard to deal with, open a port and type in https:// before the IP. Then did what the error message instructed.

Tried again...
dbvisit VISITD worked now, nemas problemas.
time to try a switchover!

Originally Posted by LINUX

Checkpoint 4 completed. Key found on dbvisit2
Copying new archives for VISITD to dbvisit2...
Compressing thread_1_seq_64.282.775132193...
Transferring thread_1_seq_64.282.775132193.gz to host
dbvisit2:thread_1_seq_64.282.775132193.gz
Compressing thread_1_seq_65.283.775132253...
Transferring thread_1_seq_65.283.775132253.gz to host
dbvisit2:thread_1_seq_65.283.775132253.gz
Transferring X.dbvisit.20120213A.VISITD.archives to host
dbvisit2:X.dbvisit.20120213A.VISITD.archives
Completed.
Shutting down regular Database VISITD...
Regular Database VISITD shutdown successfully.
Copying redo logs ... this may take a while...
201202131028 - ORA-15056: additional error messageORA-27040: file create
error, unable to create fileORA-06512: at "SYS.X$DBMS_DISKGROUP", line
253ORA-06512: at line 19
Switchover has NOT completed successfully on dbvisit1 for checkpoint 4.
Status: Database VISITD is still Primary Database.
Primary database is not open.

Rollback action on dbvisit1:
Please restart primary database with command:
dbv_oraStartStop restart VISITD

Dbvisit Standby terminated.
Return code = 433

(Tracefile required if contacting Dbvisit Standby support:
/usr/tmp/32154_dbv_oraStartStop_switchover_VISITD_201202131 028.trc

Logfile on it's way & trying the work-around you posted.


Doing the work-around:
ran into an issue when starting up the databases again after the failed switchover.

Originally Posted by LINUX

[oracle@dbvisit1 etc]$ dbv_oraStartStop restart VISITD
================================================== ===========
Dbvisit Standby Database Technology (6.0.18.8310) (pid 3077)
dbv_oraStartStop started on dbvisit1: Mon Feb 13 10:59:38 2012 (CLI)
================================================== ===========

Database VISITD on dbvisit1 is already down. No action taken.
Starting Regular Database VISITD...
Regular Database VISITD started .

================================================== ===========
dbv_oraStartStop ended on dbvisit1: Mon Feb 13 11:00:01 2012
================================================== ===========

You have new mail in /var/spool/mail/oracle
[oracle@dbvisit1 etc]$
================================================== ===========
Dbvisit Standby Database Technology
Variable ORACLE_SID_ASM_DEST is declared twice
under different categories
in Dbvisit Database configuration (DDC) file:
/usr/local/dbvisit/standby/dbv_VISITA.env
Dbvisit Standby cannot continue.
Please edit /usr/local/dbvisit/standby/dbv_VISITA.env and remove duplicates.
Exiting.
================================================== ===========

And, yes, it was stated twice again in the DDC. So a NEW line with ORACLE_SID_ASM_DEST appeared! # added. I'll send a copy of the file to you later today.

Transferring logs and applying them works nicely.

Now onto emptying the /usr/tmp/GS/VISITD directories and swtiching over:
No errors, works fine. Applying some new logs from the new primary to the new standby: no problems.
Switching back to the old primary: problem!

Originally Posted by LINUX

[oracle@dbvisit2 dbvisit]$ dbv_oraStartStop switchover VISITD 20120213D
================================================== ===========
Dbvisit Standby Database Technology (6.0.18.8310) (pid 18089)
dbv_oraStartStop started on dbvisit2: Mon Feb 13 12:40:42 2012 (CLI)
================================================== ===========

================================================== ===========
Graceful Switchover starting on Primary Database VISITD with standby
database VISITD.
Timestamp: 201202131240.

>>> Database VISITD will be shutdown and restarted <<<
Ensure Dbvisit is no longer scheduled.
201202131240 - Oracle parameter(s) db_file_name_convert and/or
log_file_name_convert set to non null value(s):
log_file_name_convert = +DATA/visitd, +DATA/VISITD/onlinelog

To proceed with switchover please follow the steps:
If using pfile, remove or comment out parameters db_file_name_convert and
log_file_name_convert in the pfile and restart the database.
If using spfile:
1) Create a pfile from the spfile in a temporary location:
SQL>create pfile='/tmp/pfile' from spfile;
2) Shut the database down
3) Edit /tmp/pfile to remove or comment out parameters db_file_name_convert
and log_file_name_convert
4) Recreate spfile:
SQL>create spfile from pfile='/tmp/pfile';
5) Restart the database
Parameters db_file_name_convert and log_file_name_convert can be reset to
original values on completion of Graceful Switchover if required.

Dbvisit Standby terminated.
Return code = 975

(Tracefile required if contacting Dbvisit Standby support:
/usr/tmp/18089_dbv_oraStartStop_switchover_VISITD_201202131 240.trc
(server:dbvisit2))

Removing parameters, restarting database, running dbvisit VISITD on both. Time for another switchover attempt... this time without emptying the /usr/tmp/GS/VISITD directory...
no problems.
a couple of dbvisit VISITD, then another switchover without emptying /usr/tmp/GS/VISITD:
Qapla'! seems to be working correctly now.


The update created another, related, problem:
On the non-ASM dstabase VISITA the problem of ASM-parameters being declared twice in the DDC appeared with the update. I #-out the ones that actually declared a parameter since VISITA isn't ASM. After that it worked like it should.