Netbackup Status code 6解决思路

我们中心系统使用nbu进行备份管理，今日发现有备份任务失败，发现了status 6的错误，下面是本人简单排查的一个过程

可以查看一下nbu备份的详细信息

记得在赛门铁克的nbu手册里面有一段记录是这样的：

那么我们可以查看我们的Oracle主机里备份脚本所输出的log日志进行下一步的诊断~

jszwdb01 [/]$ cd /oracle/nbu_scripts

jszwdb01 [/oracle/nbu_scripts]$ ls

hot_database_backup.sh           hot_database_backup_arch.sh

hot_database_backup.sh.out       hot_database_backup_arch.sh.out

jszwdb01 [/oracle/nbu_scripts]$ tail -n 200 hot_database_backup.sh.out

input datafile fno=00043 name=/dev/rjs_settindex12

input datafile fno=00089 name=/dev/rjs_settle46

input datafile fno=00135 name=/dev/rjs_statdata031

input datafile fno=00181 name=/dev/rjs_statdata077

input datafile fno=00221 name=/dev/rjs_statindex018

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34031_1_793845665 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:11

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00044 name=/dev/rjs_settindex13

input datafile fno=00090 name=/dev/rjs_settle47

input datafile fno=00136 name=/dev/rjs_statdata032

input datafile fno=00182 name=/dev/rjs_statdata078

input datafile fno=00222 name=/dev/rjs_statindex019

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34032_1_793845711 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00045 name=/dev/rjs_settle02

input datafile fno=00091 name=/dev/rjs_settle48

input datafile fno=00137 name=/dev/rjs_statdata033

input datafile fno=00183 name=/dev/rjs_statdata079

input datafile fno=00223 name=/dev/rjs_statindex020

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34033_1_793845736 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00046 name=/dev/rjs_settle03

input datafile fno=00092 name=/dev/rjs_settle49

input datafile fno=00138 name=/dev/rjs_statdata034

input datafile fno=00184 name=/dev/rjs_statdata080

input datafile fno=00002 name=/dev/rjs_undotbs01

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34034_1_793845772 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00047 name=/dev/rjs_settle04

input datafile fno=00093 name=/dev/rjs_settle50

input datafile fno=00139 name=/dev/rjs_statdata035

input datafile fno=00185 name=/dev/rjs_statdata081

input datafile fno=00224 name=/dev/rjs_undotbs02

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34035_1_793845797 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:11

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00005 name=/dev/rjs_commdata001

input datafile fno=00051 name=/dev/rjs_settle08

input datafile fno=00097 name=/dev/rjs_settle54

input datafile fno=00143 name=/dev/rjs_statdata039

input datafile fno=00001 name=/dev/rjs_system

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34036_1_793845833 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:11

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00006 name=/dev/rjs_settindex01

input datafile fno=00052 name=/dev/rjs_settle09

input datafile fno=00098 name=/dev/rjs_settle55

input datafile fno=00144 name=/dev/rjs_statdata040

input datafile fno=00003 name=/dev/rjs_sysaux

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34037_1_793845869 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00007 name=/dev/rjs_settle01

input datafile fno=00053 name=/dev/rjs_settle10

input datafile fno=00099 name=/dev/rjs_settle56

input datafile fno=00145 name=/dev/rjs_statdata041

input datafile fno=00226 name=/dev/rjs_sysaux01

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34038_1_793845904 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00008 name=/dev/rjs_statdata001

input datafile fno=00054 name=/dev/rjs_settle11

input datafile fno=00100 name=/dev/rjs_settle57

input datafile fno=00146 name=/dev/rjs_statdata042

input datafile fno=00227 name=/dev/rjs_sysaux02

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34039_1_793845930 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00009 name=/dev/rjs_statindex001

input datafile fno=00055 name=/dev/rjs_settle12

input datafile fno=00101 name=/dev/rjs_settle58

input datafile fno=00147 name=/dev/rjs_statdata043

input datafile fno=00225 name=/dev/rjs_rman

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34040_1_793845965 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00010 name=/dev/rjs_commindex01

input datafile fno=00056 name=/dev/rjs_settle13

input datafile fno=00102 name=/dev/rjs_settle59

input datafile fno=00148 name=/dev/rjs_statdata044

input datafile fno=00004 name=/dev/rjs_user

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34041_1_793845991 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00048 name=/dev/rjs_settle05

input datafile fno=00094 name=/dev/rjs_settle51

input datafile fno=00140 name=/dev/rjs_statdata036

input datafile fno=00186 name=/dev/rjs_statdata082

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34042_1_793846027 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

input datafile fno=00049 name=/dev/rjs_settle06

input datafile fno=00095 name=/dev/rjs_settle52

input datafile fno=00141 name=/dev/rjs_statdata037

input datafile fno=00187 name=/dev/rjs_statdata083

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34043_1_793846052 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

input datafile fno=00050 name=/dev/rjs_settle07

input datafile fno=00096 name=/dev/rjs_settle53

input datafile fno=00142 name=/dev/rjs_statdata038

input datafile fno=00188 name=/dev/rjs_statdata084

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34044_1_793846088 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch00: starting incremental level 1 datafile backupset

channel ch00: specifying datafile(s) in backupset

including current control file in backupset

channel ch00: starting piece 1 at 13-SEP-12

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34045_1_793846113 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:01

channel ch01: starting incremental level 1 datafile backupset

channel ch01: specifying datafile(s) in backupset

including current SPFILE in backupset

channel ch01: starting piece 1 at 13-SEP-12

channel ch00: finished piece 1 at 13-SEP-12

piece handle=bk_34046_1_793846149 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:01

channel ch01: finished piece 1 at 13-SEP-12

piece handle=bk_34047_1_793846175 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:00

Finished backup at 13-SEP-12

sql statement: alter system archive log current

released channel: ch00

released channel: ch01

allocated channel: ch00

channel ch00: sid=5693 devtype=SBT_TAPE

channel ch00: Veritas NetBackup for Oracle - Release 7.0 (2010070805)

sent command to channel: ch00

Starting backup at 13-SEP-12

current log archived

released channel: ch00

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of backup command at 09/13/2012 00:50:54

RMAN-06059: expected archived log not found, lost of archived log compromises recoverability

ORA-19625: error identifying file /ora_arch/1_143570_734312913.arc

ORA-27037: unable to obtain file status

IBM AIX RISC System/6000 Error: 2: A file or directory in the path name does not exist.

Additional information: 3

RMAN> RMAN> 

Recovery Manager complete.

Script /oracle/nbu_scripts/hot_database_backup.sh

==== ended in error on Thu Sep 13 00:50:55 GMT+08:00 2012 ====

发现了大量的RMAN备份error，其中我们关注一下ora-19625

jszwdb01 [/oracle/nbu_scripts]$ cd /ora_arch

jszwdb01 [/ora_arch]$ ls

1_144167_734312913.arc     1_144179_734312913.arc     1_144191_734312913.arc     1_144203_734312913.arc     1_144215_734312913.arc     1_144227_734312913.arc

1_144168_734312913.arc     1_144180_734312913.arc     1_144192_734312913.arc     1_144204_734312913.arc     1_144216_734312913.arc     addmrpt_1_14314_14316.txt

1_144169_734312913.arc     1_144181_734312913.arc     1_144193_734312913.arc     1_144205_734312913.arc     1_144217_734312913.arc     awrrpt_1_13612_13625.html

1_144170_734312913.arc     1_144182_734312913.arc     1_144194_734312913.arc     1_144206_734312913.arc     1_144218_734312913.arc     awrrpt_1_13614_13617.html

1_144171_734312913.arc     1_144183_734312913.arc     1_144195_734312913.arc     1_144207_734312913.arc     1_144219_734312913.arc     awrrpt_1_13618_13621.html

1_144172_734312913.arc     1_144184_734312913.arc     1_144196_734312913.arc     1_144208_734312913.arc     1_144220_734312913.arc     full_str.dump

1_144173_734312913.arc     1_144185_734312913.arc     1_144197_734312913.arc     1_144209_734312913.arc     1_144221_734312913.arc     jzjs_ora_4588116.trc

1_144174_734312913.arc     1_144186_734312913.arc     1_144198_734312913.arc     1_144210_734312913.arc     1_144222_734312913.arc     lost+found

1_144175_734312913.arc     1_144187_734312913.arc     1_144199_734312913.arc     1_144211_734312913.arc     1_144223_734312913.arc     ora_back_bitmap

1_144176_734312913.arc     1_144188_734312913.arc     1_144200_734312913.arc     1_144212_734312913.arc     1_144224_734312913.arc

1_144177_734312913.arc     1_144189_734312913.arc     1_144201_734312913.arc     1_144213_734312913.arc     1_144225_734312913.arc

1_144178_734312913.arc     1_144190_734312913.arc     1_144202_734312913.arc     1_144214_734312913.arc     1_144226_734312913.arc

jszwdb01 [/ora_arch]$ ls 

jszwdb01 [/ora_arch]$ ls 1_143570_734312913.arc

ls: 0653-341 The file 1_143570_734312913.arc does not exist.

原因找到了，操作系统下删除了这个归档日志，这个日志还没备份过，rman去备份的时候，找不到这个归档就出错了，Oracle在controlfile中记录着archivelog信息，当我们把这些物理文件delete掉或异常变动后，在controlfile中仍然记录着这些archivelog的信息，当我们手工清除archive目录下的文件后，这些记录并没有被我们从controlfile中清除掉，此时Oracle并不知道这些文件已经不存在。我们进入RMAN中执行crosscheck archivelog all；进行交叉检验，让oracle控制文件同步归档文件的信息。