Asm Dd 10M导致System文件部分坏块修复---惜分飞-尧图网站建设

📅 发布时间：2026/7/3 7:28:26

有客户的数据库由于不当操作导致asm磁盘头损坏,进行的操作命令类似

ddif=/dev/zeroof=/dev/dm-29bs=1024K count=10

asm磁盘组无法mount,提示ORA-15042

SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {1:1712:2} */

2026-05-19T21:57:16.517284-04:00

NOTE: cache registered group DATA 5/0xAE103296

NOTE: cache beganmount(first) of group DATA 5/0xAE103296

NOTE: Assigning number (5,10) to disk (/dev/dm-29)

…………

NOTE: Assigning number (5,18) to disk (/dev/dm-15)

NOTE: Assigning number (5,0) to disk (/dev/dm-7)

NOTE: Assigning number (5,15) to disk (/dev/dm-5)

2026-05-19T21:57:16.650481-04:00

cluster guid (b216d47e2bf86f6aff34a36119b0c161) generatedforPST Hbeatforinstance 1

2026-05-19T21:57:22.659262-04:00

NOTE: GMON heartbeatingforgrp 5 (DATA)

GMON querying group 5 at 28forpid 42, osid 12996

2026-05-19T21:57:22.661447-04:00

NOTE: Assigning number (5,14) to disk ()

2026-05-19T21:57:22.662476-04:00

GMON querying group 5 at 29forpid 42, osid 12996

2026-05-19T21:57:22.663144-04:00

NOTE: cache dismounting (clean) group 5/0xAE103296(DATA)

NOTE: messaging CKPT to quiesce pins Unix process pid: 12996, image: oracle@dlycdb1 (TNS V1-V3)

NOTE: dbwr not being msg'd to dismount

NOTE: LGWR not being messaged to dismount

NOTE: cache dismounted group 5/0xAE103296(DATA)

NOTE: cache endingmount(fail) of group DATA number=5 incarn=0xae103296

NOTE: cache deleting contextforgroup DATA 5/0xae103296

2026-05-19T21:57:22.720673-04:00

GMON dismounting group 5 at 30forpid 42, osid 12996

2026-05-19T21:57:22.721055-04:00

NOTE: Disk DATA_0000inmode 0x7f markedforde-assignment

NOTE: Disk DATA_0001inmode 0x7f markedforde-assignment

…………

NOTE: Disk DATA_0013inmode 0x7f markedforde-assignment

NOTE: Disk DATA_0015inmode 0x7f markedforde-assignment

NOTE: Disk DATA_0016inmode 0x7f markedforde-assignment

NOTE: Disk DATA_0017inmode 0x7f markedforde-assignment

NOTE: Disk DATA_0018inmode 0x7f markedforde-assignment

ERROR: diskgroup DATA was not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk"14"is missing from group number"5"

2026-05-19T21:57:22.745140-04:00

ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {1:1712:2} */

由于客户这个是19c版本,直接使用备份au还原,然后mount磁盘组成功

SQL> alter diskgroup datamount

2026-05-19T22:40:59.137657+08:00

NOTE: cache registered group DATA 2/0xB126DA41

NOTE: cache beganmount(first) of group DATA 2/0xB126DA41

NOTE: Assigning number (2,14) to disk (/dev/dm-29)

NOTE: Assigning number (2,8) to disk (/dev/dm-28)

…………

NOTE: Assigning number (2,2) to disk (/dev/dm-7)

NOTE: Assigning number (2,0) to disk (/dev/dm-6)

NOTE: Assigning number (2,1) to disk (/dev/dm-3)

2026-05-19T22:40:59.303908+08:00

cluster guid (b216d47e2bf86f6aff34a36119b0c161) generatedforPST Hbeatforinstance 2

2026-05-19T22:41:05.312792+08:00

NOTE: GMON heartbeatingforgrp 2 (DATA)

GMON querying group 2 at 80forpid 35, osid 53051

2026-05-19T22:41:05.314925+08:00

NOTE: cache is mounting group DATA created on 2024/11/2817:55:45

NOTE: cache opening disk 0 of grp 2: DATA_0000 path:/dev/dm-6

NOTE: 05/20/2616:41:04 DATA.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1

NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/dm-3

…………

NOTE: cache opening disk 13 of grp 2: DATA_0013 path:/dev/dm-18

NOTE: cache opening disk 14 of grp 2: DATA_0014 path:/dev/dm-29

NOTE: cache opening disk 15 of grp 2: DATA_0015 path:/dev/dm-16

NOTE: cache opening disk 16 of grp 2: DATA_0016 path:/dev/dm-17

NOTE: cache opening disk 17 of grp 2: DATA_0017 path:/dev/dm-20

NOTE: cache opening disk 18 of grp 2: DATA_0018 path:/dev/dm-21

2026-05-19T22:41:05.317307+08:00

NOTE: cache mounting (first) external redundancy group 2/0xB126DA41(DATA)

2026-05-19T22:41:05.522191+08:00

NOTE: attached to recovery domain 2

2026-05-19T22:41:05.558136+08:00

validate pdb 2, flags x4, valid 0, pdb flags x204

* validated domain 2, flags = 0x200

NOTE: cache recovered group 2 to fcn 0.46336611

NOTE: redo buffer size is 512 blocks (2105344 bytes)

2026-05-19T22:41:05.569479+08:00

NOTE: LGWR attempting tomountthread 1fordiskgroup 2 (DATA)

NOTE: LGWR found thread 1 closed at ABA 110.6507 lock domain=0 inc#=0 instnum=1

NOTE: LGWR mounted thread 1fordiskgroup 2 (DATA)

2026-05-19T22:41:05.574499+08:00

mntstmp=2026/05/2016:41:05.572000

2026-05-19T22:41:05.574854+08:00

NOTE: cache mounting group 2/0xB126DA41(DATA) succeeded

NOTE: cache endingmount(success) of group DATA number=2 incarn=0xb126da41

2026-05-19T22:41:05.616227+08:00

NOTE: Instance updated compatible.asm to 19.0.0.0.0forgrp 2 (DATA).

2026-05-19T22:41:05.616754+08:00

NOTE: Instance updated compatible.asm to 19.0.0.0.0forgrp 2 (DATA).

2026-05-19T22:41:05.617932+08:00

NOTE: Instance updated compatible.rdbms to 19.0.0.0.0forgrp 2 (DATA).

2026-05-19T22:41:05.618303+08:00

NOTE: Instance updated compatible.rdbms to 19.0.0.0.0forgrp 2 (DATA).

2026-05-19T22:41:05.643844+08:00

SUCCESS: diskgroup DATA was mounted

虽然磁盘组mount成功,但是asm依旧在报错

这个客户胆子也真够大的,这样mount起来的磁盘组,还往里面加入磁盘出发Rebalance操作

1

2026-05-20T09:18:34.292103-04:00

Errorsinfile/u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1014] [0 != 1]

NOTE: cache repaired a corrupt block: group=2(DATA) dsk=14 blk=1014 on disk 14 from disk=14 (DATA_0014) incarn=4042690154 au=11 blk=1014 count=1

2026-05-20T09:18:36.721669-04:00

WARNING: cachereada corrupt block: group=2(DATA) dsk=14 blk=1015 disk=14 (DATA_0014) incarn=4042690154 au=0 blk=1015 count=1

2026-05-20T09:18:36.721982-04:00

Errorsinfile/u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1015] [0 != 1]

NOTE: a corrupted block from group DATA was dumped to/u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc

WARNING: cacheread(retry) a corrupt block: group=2(DATA) dsk=14 blk=1015 disk=14 (DATA_0014) incarn=4042690154 au=0 blk=1015 count=1

2026-05-20T09:18:36.724279-04:00

Errorsinfile/u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_60220.trc:

ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] [2147483662] [1015] [0 != 1]

NOTE: cache repaired a corrupt block: group=2(DATA) dsk=14 blk=1015 on disk 14 from disk=14 (DATA_0014) incarn=4042690154 au=11 blk=1015 count=1

2026-05-20T09:44:20.469792-04:00

NOTE: Starting expel slaveforgroup 2/0xcd67eb4(DATA)

2026-05-20T09:44:20.473880-04:00

NOTE: GroupBlock outside rolling migration privileged region

NOTE: requesting all-instance membership refreshforgroup=2

2026-05-20T09:44:20.530101-04:00

NOTE: membership refresh pendingforgroup 2/0xcd67eb4(DATA)

2026-05-20T09:44:20.534097-04:00

GMON querying group 2 at 178forpid 27, osid 70130

2026-05-20T09:44:20.557651-04:00

SUCCESS: refreshed membershipfor2/0xcd67eb4(DATA)

2026-05-20T09:44:23.489304-04:00

NOTE: Attempting votingfilerefresh on diskgroup DATA

NOTE: Refresh completed on diskgroup DATA. No votingfilefound.

2026-05-20T11:20:49.804413-04:00

NOTE: stopping process ARB0

NOTE: stopping process ARBA

2026-05-20T11:20:51.538777-04:00

SUCCESS: rebalance completedforgroup 2/0xcd67eb4(DATA)

2026-05-20T23:20:51.541462+08:00

SUCCESS: ALTER DISKGROUP DATA ADD DISK'/dev/dm-20'SIZE 4194304M

REBALANCE WAIT

算幸运,由于ORA-15196: invalid ASM block header [kfc.c:30747] [endian_kfbh] 错误导致rebalance没有真正运行起来,从而该磁盘组没有dismount(19c这个方面确实增强不少,如果以前版本大概率会直接dismount掉)

客户在这样mount的磁盘组上尝试启动库,报ORA-01578错误,无法启动成功

2026-05-20T17:46:07.060794+08:00

Error attempting to elevate LMHB's priority: no further priority changes will be attemptedforthis process

2026-05-20T17:46:07.647114+08:00

Undo initialization recovery: Parallel FPTR complete: start:26091908 end:26096229diff:4321 ms (4.3 seconds)

Undo initialization recovery: err:0 start: 26091907 end: 26096229diff: 4322 ms (4.3 seconds)

[50880] Successfully onlined Undo Tablespace 5.

Undo initialization online undo segments: err:0 start: 26096229 end: 26096576diff: 347 ms (0.3 seconds)

Undo initialization finished serial:0 start:26091907 end:26096591diff:4684 ms (4.7 seconds)

Database Characterset is AL32UTF8

No Resource Manager plan active

2026-05-20T17:46:08.734113+08:00

Corrupt block relative dba: 0x004030ee (file1, block 12526)

Completely zero block found during bufferread

Reread (file1, block 12526) found same corrupt data (no logical check)

2026-05-20T17:46:08.811455+08:00

Corrupt Block Found

TIME STAMP (GMT) = 05/20/202617:46:07

CONT = 0, TSN = 0, TSNAME = SYSTEM

RFN = 1, BLK = 12526, RDBA = 4206830

OBJN = 37, OBJD = 37, OBJECT = I_OBJ2, SUBOBJECT =

SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc (incident=800708):

ORA-01578: ORACLE data block corrupted (file# 1, block # 12526)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T05:46:09.047772-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521'SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:09.049615-04:00

ALTER SYSTEM SET listener_networks=''SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:09.812271+08:00

*****************************************************************

An internal routine has requested a dump of selected redo.

This usually happens following a specific internal error, when

analysis of the redo logs will help Oracle Support with the

diagnosis.

It is recommended that you retain all the redo logs generated (by

all the instances) during the past 12 hours,incaseadditional

redo dumps are required to help with the diagnosis.

*****************************************************************

Corrupt block relative dba: 0x004030de (file1, block 12510)

Completely zero block found during bufferread

Reread (file1, block 12510) found same corrupt data (no logical check)

2026-05-20T17:46:10.350573+08:00

Corrupt Block Found

TIME STAMP (GMT) = 05/20/202617:46:09

CONT = 0, TSN = 0, TSNAME = SYSTEM

RFN = 1, BLK = 12510, RDBA = 4206814

OBJN = 83, OBJD = 83, OBJECT = DEPENDENCY$, SUBOBJECT =

SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc (incident=800709):

ORA-01578: ORACLE data block corrupted (file# 1, block # 12510)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:10.694365+08:00

Corrupt block relative dba: 0x00403000 (file1, block 12288)

Completely zero block found during validation

Reread of blocknum=12288,file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Corrupt block relative dba: 0x00403001 (file1, block 12289)

Completely zero block found during validation

Reread of blocknum=12289,file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

………………

Corrupt block relative dba: 0x004030ff (file1, block 12543)

Completely zero block found during validation

Reread of blocknum=12543,file=+DATA/xff/DATAFILE/system.257.1186720165. found same corrupt data

Corrupt block relative dba: 0x0040301d (file1, block 12317)

Completely zero block found during bufferread

Reread (file1, block 12317) found same corrupt data (no logical check)

2026-05-20T17:46:12.183545+08:00

Corrupt Block Found

TIME STAMP (GMT) = 05/20/202617:46:11

CONT = 0, TSN = 0, TSNAME = SYSTEM

RFN = 1, BLK = 12317, RDBA = 4206621

OBJN = 37, OBJD = 37, OBJECT = I_OBJ2, SUBOBJECT =

SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc (incident=800710):

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

Incident detailsin:/u01/app/oracle/diag/rdbms/xff/xff2/incident/incdir_800710/xff2_ora_50880_i800710.trc

2026-05-20T05:46:12.371040-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521'SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:12.372924-04:00

ALTER SYSTEM SET listener_networks=''SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:13.814346+08:00

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814407+08:00

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814442+08:00

Error 604 happened during dbopen, shutting down database

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc (incident=800711):

ORA-00603: ORACLE server session terminated by fatal error

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

Corrupt block relative dba: 0x00403097 (file1, block 12439)

Completely zero block found during bufferread

Reread (file1, block 12439) found same corrupt data (no logical check)

2026-05-20T17:46:14.259044+08:00

Corrupt Block Found

TIME STAMP (GMT) = 05/20/202617:46:13

CONT = 0, TSN = 0, TSNAME = SYSTEM

RFN = 1, BLK = 12439, RDBA = 4206743

OBJN = 18, OBJD = 18, OBJECT = OBJ$, SUBOBJECT =

SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_gen0_48604.trc (incident=798852):

ORA-01578: ORACLE data block corrupted (file# 1, block # 12439)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814346+08:00

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814407+08:00

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc:

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T17:46:13.814442+08:00

Error 604 happened during dbopen, shutting down database

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_50880.trc (incident=800711):

ORA-00603: ORACLE server session terminated by fatal error

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00604: error occurred at recursive SQL level 1

ORA-01578: ORACLE data block corrupted (file# 1, block # 12317)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

Corrupt block relative dba: 0x00403097 (file1, block 12439)

Completely zero block found during bufferread

Reread (file1, block 12439) found same corrupt data (no logical check)

2026-05-20T17:46:14.259044+08:00

Corrupt Block Found

TIME STAMP (GMT) = 05/20/202617:46:13

CONT = 0, TSN = 0, TSNAME = SYSTEM

RFN = 1, BLK = 12439, RDBA = 4206743

OBJN = 18, OBJD = 18, OBJECT = OBJ$, SUBOBJECT =

SEGMENT OWNER = SYS, SEGMENT TYPE = Table Segment

Errorsinfile/u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_gen0_48604.trc (incident=798852):

ORA-01578: ORACLE data block corrupted (file# 1, block # 12439)

ORA-01110: datafile1:'+DATA/xff/DATAFILE/system.257.1186720165'

2026-05-20T05:46:14.436597-04:00

ALTER SYSTEM SET remote_listener=' xff-scan:11521'SCOPE=MEMORY SID='xff2';

2026-05-20T05:46:14.438492-04:00

ALTER SYSTEM SET listener_networks=''SCOPE=MEMORY SID='xff2';

2026-05-20T17:46:15.486758+08:00

opiodr aborting process unknown ospid (50880) as a result of ORA-603

2026-05-20T17:46:15.498707+08:00

ORA-603 : opitsk aborting process

License high water mark = 423

USER(prelim) (ospid: 50880): terminating the instance due to ORA error 604

2026-05-20T17:46:15.536740+08:00

opiodr aborting process unknown ospid (69547) as a result of ORA-1092

2026-05-20T05:46:16.321597-04:00

ORA-1092 : opitsk aborting process

这里基本上可以看出来是由于在数据库启动过程中递归调用一些sql,但是由于遭遇到坏块导致启动失败,通过dbv检查system数据文件发现256个坏块

256个连续的全0坏块,怀疑是2M的数据被dd全空覆盖,这样的情况,也就是怀疑是au=2的后面2M被覆盖(ausize为4M),分析system的数据分布情况

这里可以确认system的第24个au(从0开始)在14号盘au 2 上面,也就是数据块起始损坏为block：12288-12543（24M*4/8K[有block 0 需要考虑])，对于这种彻底损坏而且比较靠前的system中block,通过人工构造出来这些block的方式进行修复,在自研的Oracle Recovery Tools和obet工具都有该功能.运气不错,通过这个修复之后,直接expdp导出数据没有大问题,比较完美的恢复了这个故障.