ORA-15040 ora-15017ASM磁盘无法挂载故障处理

秒速五厘米 2023-01-19 12:49 17阅读 0赞

测试环境突然一个节点的数据库无法启动，报错很清楚，找不到控制文件，而控制文件在ASM上，第一个节点正常，下面是处理流程。  
数据库启动报错  
SQL> startup  
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance  
ORACLE instance started.

Total System Global Area 2684354560 bytes  
Fixed Size 8624312 bytes  
Variable Size 989857608 bytes  
Database Buffers 1677721600 bytes  
Redo Buffers 8151040 bytes  
ORA-00205: error in identifying control file, check alert log for more info

告警日志：  
alter database mount  
2021-04-22T15:06:27.293867+08:00  
NOTE: ASMB mounting group 3 (USERDATA)  
NOTE: ASM background process initiating disk discovery for grp 3 (reqid:0)  
WARNING: group 3 (USERDATA) has missing disks  
ORA-15040: diskgroup is incomplete  
WARNING: group 3 is being dismounted.  
WARNING: ASMB force dismounting group 3 (USERDATA) due to missing disks  
SUCCESS: diskgroup USERDATA was dismounted  
2021-04-22T15:06:27.330633+08:00  
ORA-00210: cannot open the specified control file  
ORA-00202: control file: ‘+USERDATA/PROD/CONTROLFILE/current.257.1055783219’  
ORA-17503: ksfdopn:2 Failed to open file +USERDATA/PROD/CONTROLFILE/current.257.1055783219  
ORA-15001: diskgroup “USERDATA” does not exist or is not mounted  
ORA-15040: diskgroup is incomplete  
ORA-205 signalled during: alter database mount…

磁盘组状态  
SQL> select name,state,total\_mb,type from v$asm\_diskgroup

NAME STATE TOTAL\_MB TYPE

--------------------

ACFSDATA MOUNTED 12224 EXTERN  
DATA MOUNTED 51196 EXTERN  
USERDATA MOUNTED 20476 EXTERN

查看磁盘组的状态  
SQL>select b.name,path,a.state,a.mount\_status,a.header\_status from v a s m d i s k a , v asm\_disk a,v asmdiska,vasm\_diskgroup b where a.group\_number=b.group\_number

NAME PATH STATE MOUNT\_STATUS HEADER\_STATUS

--------------------

USERDATA /dev/sdd1 NORMAL CACHED MEMBER  
ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER  
DATA /dev/sdc1 NORMAL CACHED MEMBER

查看磁盘信息

Disk /dev/sdd: 21.5 GB, 21474836480 bytes, 41943040 sectors  
Units = sectors of 1 \* 512 = 512 bytes  
Sector size (logical/physical): 512 bytes / 512 bytes  
I/O size (minimum/optimal): 512 bytes / 512 bytes  
Disk label type: dos  
Disk identifier: 0x7bf56a92

Device Boot Start End Blocks Id System  
/dev/sdd1 2048 41943039 20970496 83 Linux

尝试重新mount磁盘组  
SQL> alter diskgroup usedata mount;  
alter diskgroup usedata mount  
\*  
ERROR at line 1:  
ORA-15032: not all alterations performed  
ORA-15017: diskgroup “USEDATA” cannot be mounted  
ORA-15040: diskgroup is incomplete

下面尝试mount正常的磁盘组  
SQL> alter diskgroup ACFSDATA mount;  
alter diskgroup ACFSDATA mount  
\*  
ERROR at line 1:  
ORA-15032: not all alterations performed  
ORA-15017: diskgroup “ACFSDATA” cannot be mounted  
ORA-15013: diskgroup “ACFSDATA” is already mounted

显然二者的提示不同，问题磁盘组报错15040，下马是是该错误解释  
\[oracle@rac2 ~\]$ oerr ora 15040  
15040, 00000, “diskgroup is incomplete”  
// \*Cause: Some of the disks comprising a diskgroup were not present.  
// \*Action: Check the hardware to ensure that all disks are functional. Also  
// check that the setting of the ASM\_DISKSTRING initialization  
// parameter has not changed. Alternatively, for normal or high  
// redundancy diskgroups, use MOUNT FORCE to offline missing disks  
// as part of mounting the diskgroup.

找不到ASM磁盘了，但是第一个节点该磁盘组是正常的，因为只有一个ASM磁盘，我们排除是ASM磁盘头的问题  
应该是第二个节点与操作系统之间有问题，也就是实例2无法识别这个ASM磁盘

节点2 userdata磁盘组dismount后的状态  
SQL> l  
1\* select b.name,path,a.state,a.mount\_status,a.header\_status from v a s m d i s k a , v asm\_disk a,v asmdiska,vasm\_diskgroup b where a.group\_number=b.group\_number  
SQL> /

NAME PATH STATE MOUNT\_STATUS HEADER\_STATUS

--------------------

USERDATA /dev/sde1 NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdg1 NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdg NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdb NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdc NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdf NORMAL CLOSED CANDIDATE  
USERDATA /dev/sde NORMAL CLOSED CANDIDATE  
USERDATA /dev/sda2 NORMAL CLOSED CANDIDATE  
USERDATA /dev/sdd NORMAL CLOSED CANDIDATE  
USERDATA /dev/sda1 NORMAL CLOSED CANDIDATE  
ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER  
DATA /dev/sdc1 NORMAL CACHED MEMBER

12 rows selected.  
节点2多了很多candidate盘，并且磁盘组userdata没有/dev/sdd1的盘为member状态。  
节点1 userdata磁盘组dismount后的状态（该节点正常）  
select 'alter diskgroup ‘||b.name||’ drop disk '||path from v a s m d i s k a , v asm\_disk a,v asmdiska,vasm\_diskgroup b where a.group\_number=b.group\_number;

SQL> select b.name,path,a.state,a.mount\_status,a.header\_status from v a s m d i s k a , v asm\_disk a,v asmdiska,vasm\_diskgroup b where a.group\_number=b.group\_number

NAME PATH STATE MOUNT\_STATUS HEADER\_STATUS

--------------------

USERDATA /dev/sdd1 NORMAL CLOSED MEMBER  
USERDATA /dev/sdf1 NORMAL CLOSED FORMER  
ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER  
DATA /dev/sdc1 NORMAL CACHED MEMBER

显然两个节点对应的设备不同，从candidate含义可以知道多了备选的ASM磁盘，于是查询rules规则文件  
\[oracle@rac2 rules.d\]$ ls -lrt *asm*  
\-rw-r–r-- 1 root root 712 Nov 1 13:16 99-my-asmdevices.rules  
\-rw-r–r-- 1 root root 1684 Apr 20 17:09 99-oracle-asmdevices.rules

发现节点2有两个规则，文件，突然记起来之前测试了一个自动生成rules文件的脚本，脚本内容如下，也就解释了为何出现这个问题。  
\[oracle@rac2 rules.d\]$ cat 99-oracle-asmdevices.rules  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VBf0c224c2-df949031”, SYMLINK+=“asm\_sdb”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VB6696fa7f-9e3b849f”, SYMLINK+=“asm\_sdc”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VBd23c9420-9c9f27e4”, SYMLINK+=“asm\_sdd”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VBfd0b0c16-d27f68c4”, SYMLINK+=“asm\_sde”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VBa7ed9674-7578ffc6”, SYMLINK+=“asm\_sdf”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT==“1ATA\_VBOX\_HARDDISK\_VBed0c7e0c-483cfad3”, SYMLINK+=“asm\_sdg”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d $devnode",  
RESULT=="", SYMLINK+=“asm\_sdh”, OWNER=“grid”, GROUP=“asmadmin”, MODE=  
“0660”  
KERNEL==“sd\*”, ENV\{DEVTYPE\}“disk”, SUBSYSTEM"block", PROGRAM=="/lib/udev/scsi\_id -g -u -d  d e v n o d e " , R E S U L T = = " " , S Y M L I N K + = " a s m s d i " , O W N E R = " g r i d " , G R O U P = " a s m a d m i n " , M O D E = " 0660 " 问 题 好 解 决 了 ， 删 除 最 后 的 那 个 规 则 文 件 再 次 查 询 节 点 2 S Q L > l 1 ∗ s e l e c t b . n a m e , p a t h , a . s t a t e , a . m o u n t s t a t u s , a . h e a d e r s t a t u s f r o m v devnode", RESULT=="", SYMLINK+="asm\_sdi", OWNER="grid", GROUP="asmadmin", MODE= "0660" 问题好解决了，删除最后的那个规则文件 再次查询节点2 SQL> l 1\* select b.name,path,a.state,a.mount\_status,a.header\_status from v devnode",RESULT=="",SYMLINK\+="asmsdi",OWNER="grid",GROUP="asmadmin",MODE="0660"问题好解决了，删除最后的那个规则文件再次查询节点2SQL>l1∗selectb.name,path,a.state,a.mountstatus,a.headerstatusfromvasm\_disk a,v$asm\_diskgroup b where a.group\_number=b.group\_number  
SQL> /

NAME PATH STATE MOUNT\_STATUS HEADER\_STATUS

--------------------

ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER  
USERDATA /dev/sdd1 NORMAL CACHED MEMBER  
DATA /dev/sdc1 NORMAL CACHED MEMBER

下面重启下集群，重新挂载ASM磁盘组
    看节点2数据库告警日志
    ALTER DATABASE MOUNT /* db agent *//* { 2:10468:2} */
    2021-04-22T15:41:43.380715+08:00
    This instance was first to mount
    2021-04-22T15:41:44.382690+08:00
    NOTE: ASMB mounting group 3 (USERDATA)
    NOTE: ASM background process initiating disk discovery for grp 3 (reqid:0)
    NOTE: Assigning number (3,0) to disk (/dev/sdd1)
    SUCCESS: mounted group 3 (USERDATA)
    NOTE: grp 3 disk 0: USERDATA_0000 path:/dev/sdd1
    2021-04-22T15:41:44.719410+08:00
    NOTE: dependency between database prod and diskgroup resource ora.USERDATA.dg is established
    这里提示成功挂载ASM磁盘
    集群状态：

## \[oracle@rac1 ~\]$ crsctl stat res -t ##

## Name Target State Server State details ##

## Local Resources ##

ora.ACFSDATA.dg
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.ASMNET1LSNR_ASM.lsnr
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.DATA.dg
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.LISTENER.lsnr
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.USERDATA.dg
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.net1.network
    ONLINE ONLINE rac1 STABLE
    ONLINE ONLINE rac2 STABLE
    ora.ons
    ONLINE ONLINE rac1 STABLE

## ONLINE ONLINE rac2 STABLE ##

## Cluster Resources ##

ora.LISTENER_SCAN1.lsnr
    1 ONLINE ONLINE rac1 STABLE
    ora.MGMTLSNR
    1 OFFLINE OFFLINE STABLE
    ora.asm
    1 ONLINE ONLINE rac1 Started,STABLE
    2 ONLINE ONLINE rac2 Started,STABLE
    3 OFFLINE OFFLINE STABLE
    ora.cvu
    1 ONLINE ONLINE rac1 STABLE
    ora.mgmtdb
    1 OFFLINE OFFLINE STABLE
    ora.prod.db
    1 ONLINE ONLINE rac1 Open,HOME=/oracle/db
    /base/product/12.2,S
    TABLE
    2 ONLINE ONLINE rac2 Open,HOME=/oracle/db
    /base/product/12.2,S
    TABLE
    ora.qosmserver
    1 ONLINE ONLINE rac1 STABLE
    ora.rac1.vip
    1 ONLINE ONLINE rac1 STABLE
    ora.rac2.vip
    1 ONLINE ONLINE rac2 STABLE
    ora.scan1.vip

## 1 ONLINE ONLINE rac1 STABLE ##

提示：不要随便在运行库测试脚本，造成生成的规则文件覆盖之前的文件，造成一个节点的数据库无法启动。

原文链接：[http://github.crmeb.net/u/defu][http_github.crmeb.net_u_defu]

[http_github.crmeb.net_u_defu]: http://github.crmeb.net/u/defu