Switchover failure due to ASM & redo log files.

[Gustav]

We're evaluating Dbvisit Standby for a client and have run into a problem which we haven't been able to find the solution for, neither on this forum nor via the google.

Background:
Set up is on two identical machines: testhost1 and testhost2. Oracle 11.2.0.3 is running on both. TESTDB has been created with ASM on testhost1. Creating the standby db on testhost2 works, as transferring and applying logs to it. It is when attempting the switchover that an error occurs.
Last weeks testing of a non-ASM db worked like a charm.

The problem:
20120208 12:41:58 main::cmn_mail_logs-> Mail_subject:ERROR with Dbvisit Standby processing on testhost1 for TESTDB (from testhost1 to testhost2)
20120208 12:41:58 main::cmn_print-> cmn_print:
201202081238 -
ORA-19505: failed to identify file
ORA-27040: file create error, unable to create file
ORA-15120: ASM file name '/usr/tmp/GS/TESTDB/X.dbvisit.307.TESTDB.redo_1.log'
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 413ORA-06512: at line 11
Switchover has NOT completed successfully on testhost1 for checkpoint 4.
Status: Database TESTDB is still Primary Database. Primary database is not open. Rollback action on testhost1:
Please restart primary database with command: dbv_oraStartStop restart TESTDB
Dbvisit Standby terminated. Return code = 19505

How do we solve this?

[Arjen Visser]

Hello,
Can you tell us what version of Dbvisit Standby you are using?

Are you also able to copy some more of the trace file so that we can see what proceeded the error?

We need to find the cause of the error: ORA-27040: file create error, unable to create file
Is there anything in the +ASM alert log that is useful?
Is there enough free space in +ASM?

Thanks and regards

[Anna Macnaughtan]

Hello

It looks like Dbvisit failed to generate a correct filename for the new redo log: /usr/tmp/GS/TESTDB/X.dbvisit.307.TESTDB.redo_1.log is not an ASM name.
Can you please email me the whole trace file to investigate?
Thank you
anna.macnaughtan@dbvisit.com

[Gustav]

The tracefile is on its way, along with the answers to Arjen's questions.

[Anna Macnaughtan]

Hello Gustav

The error occurs when Dbvisit attempts to copy a redo log from ASM to filesystem using the dbms_diskgroup PL/SQL package while connecting to the ASM instance:

20120208 12:41:58 main::cmn_cat-> SQL> Connected.
20120208 12:41:58 main::cmn_cat-> SQL> SQL> SQL> SQL> 2 3 4 5 6 7 8 9 10 11 12 13 fileType=3 fileSz=102400 blkSz=512
20120208 12:41:58 main::cmn_cat-> declare
20120208 12:41:58 main::cmn_cat-> *
20120208 12:41:58 main::cmn_cat-> ERROR at line 1:
20120208 12:41:58 main::cmn_cat-> ORA-19505: failed to identify file
20120208 12:41:58 main::cmn_cat-> "/usr/tmp/GS/VISITD/X.dbvisit.307.VISITD.redo_1.log"
20120208 12:41:58 main::cmn_cat-> ORA-27040: file create error, unable to create file
20120208 12:41:58 main::cmn_cat-> Linux-x86_64 Error: 13: Permission denied
20120208 12:41:58 main::cmn_cat-> Additional information: 1
20120208 12:41:58 main::cmn_cat-> ORA-15120: ASM file name '/usr/tmp/GS/VISITD/X.dbvisit.307.VISITD.redo_1.log'
20120208 12:41:58 main::cmn_cat-> does not begin with the ASM prefix character
20120208 12:41:58 main::cmn_cat-> ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 413
20120208 12:41:58 main::cmn_cat-> ORA-06512: at line 11

We have not come across this error in our testing environment and have not managed to reproduce it. Can you please perform the following steps on the primary to help us investigate this further:

1. Double check directories /usr/tmp/GS/VISITD and /usr/tmp/GS exist and are writable for user oracle (we expect permissions to be drwxr-xr-x)
2. Run the following command and send us the trace file:
dbv_functions -a VISITD asm_copy_file +DATA/visitd/redo01.log=/usr/tmp/GS/VISITD/X.dbvisit.307.VISITD.redo_1.log
3. Create a directory with permissions drwxrwxrwx (say /tmp), run the following command and send us the trace:
dbv_functions -a VISITD asm_copy_file +DATA/visitd/redo01.log=/tmp/redo_1.log
4. Add the following line to the end of your DDC file:
_FORCE_ASM_READ_WRITE=Yes
and run the command from step 3 once again

Thank you for your assistance.

[Gustav]

Originally Posted by Anna Macnaughtan

We have not come across this error in our testing environment and have not managed to reproduce it. Can you please perform the following steps on the primary to help us investigate this further:

1. Double check directories /usr/tmp/GS/VISITD and /usr/tmp/GS exist and are writable for user oracle (we expect permissions to be drwxr-xr-x)

Those check out, drwxr-xr-x oracle oinstall on all of 'em.

Originally Posted by Anna Macnaughtan

2. Run the following command and send us the trace file:
dbv_functions -a VISITD asm_copy_file +DATA/visitd/redo01.log=/usr/tmp/GS/VISITD/X.dbvisit.307.VISITD.redo_1.log
3. Create a directory with permissions drwxrwxrwx (say /tmp), run the following command and send us the trace:
dbv_functions -a VISITD asm_copy_file +DATA/visitd/redo01.log=/tmp/redo_1.log
4. Add the following line to the end of your DDC file:
_FORCE_ASM_READ_WRITE=Yes
and run the command from step 3 once again

Thank you for your assistance.

I'm sending all the tracefiles in minute, but first I want to say that step 3 did not return any errors. Step 4's re-run of step 3 did, so I a also did one with a changed write-target.