oracle grid ocr磁碟組丟失恢復
版本:11.2.0.4.0
Reference : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) (Doc ID 428681.1)
CRS-4256 CRS-4602 While Replacing Voting Disk (文件 ID 1475588.1)
How to Restore ASM Based OCR After Complete Loss of the CRS Diskgroup on Linux/Unix Systems (文件 ID 1062983.1)
環境檢查:
確認ocr,votedisk 磁碟位置。
[[email protected] ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 309717e185584f38bf0380d7e53e95ee (/dev/asm-diskb) [DATA] 2. ONLINE 791a04adb2534f28bf4954cfa12289a5 (/dev/asm-diskc) [DATA] 3. ONLINE af884b0a019d4f77bf3c0269106a3c95 (/dev/asm-diskd) [DATA] Located 3 voting disk(s). [
[email protected] ~]$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2828 Available space (kbytes) : 259292 ID : 671775796 Device/File Name : +DATA Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user
模擬破壞磁碟
[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskb bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.72446 s, 608 MB/s
[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskc bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.75642 s, 597 MB/s
[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskd bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.91177 s, 548 MB/s
[[email protected] gpnpd]#
檢查叢集狀態
[[email protected] ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
重建磁碟組,
在一個節點操作,其它節點處於關閉狀態。
/u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
建立data磁碟組,修改 asm_diskstring引數。
[[email protected] ~]# /u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'prod01'
CRS-2676: Start of 'ora.mdnsd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prod01'
CRS-2676: Start of 'ora.gpnpd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prod01'
CRS-2672: Attempting to start 'ora.gipcd' on 'prod01'
CRS-2676: Start of 'ora.cssdmonitor' on 'prod01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prod01'
CRS-2672: Attempting to start 'ora.diskmon' on 'prod01'
CRS-2676: Start of 'ora.diskmon' on 'prod01' succeeded
CRS-2676: Start of 'ora.cssd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'prod01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2672: Attempting to start 'ora.ctssd' on 'prod01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2676: Start of 'ora.drivers.acfs' on 'prod01' succeeded
CRS-2676: Start of 'ora.ctssd' on 'prod01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prod01'
CRS-2676: Start of 'ora.asm' on 'prod01' succeeded
[[email protected] ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 13:59:13 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup DATA normal redundancy
DISK '/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd'
attribute 'compatible.asm'='11.2'; 2 3
Diskgroup created.
SQL> alter system set asm_diskstring='/dev/*';
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[[email protected] ~]$
恢復ocr,votedisk
ocrconfig -showbackup
root:
/u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr
grid
[[email protected] ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
prod02 2018/11/21 09:00:45 /u01/app/11.2.0/grid/cdata/prod-cluster/backup00.ocr
prod02 2018/11/21 05:00:45 /u01/app/11.2.0/grid/cdata/prod-cluster/backup01.ocr
prod02 2018/11/21 01:00:44 /u01/app/11.2.0/grid/cdata/prod-cluster/backup02.ocr
prod02 2018/11/20 01:00:42 /u01/app/11.2.0/grid/cdata/prod-cluster/day.ocr
prod02 2018/11/12 21:00:25 /u01/app/11.2.0/grid/cdata/prod-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
#root 恢復ocr
[[email protected] ~]# /u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr
#root 恢復votedisk
[[email protected] bin]# ./crsctl replace votedisk +DATA
Successful addition of voting disk 96e6e83cf3364f9abf13c9310830d798.
Successful addition of voting disk 3d317981258b4fcfbf64f119f41a7f28.
Successful addition of voting disk 3aa046d1930c4f39bf9bcf3297554d85.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
驗證叢集狀態
關閉叢集# crsctl stop crs -f
啟動所有節點叢集:# crsctl start crs
[[email protected] bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.DG1.dg
ONLINE OFFLINE prod01
ONLINE OFFLINE prod02
ora.LISTENER.lsnr
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.asm
ONLINE ONLINE prod01 Started
ONLINE ONLINE prod02 Started
ora.gsd
OFFLINE OFFLINE prod01
OFFLINE OFFLINE prod02
ora.net1.network
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.ons
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.registry.acfs
ONLINE ONLINE prod01
ONLINE ONLINE prod02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod01
ora.cvu
1 OFFLINE OFFLINE
ora.oc4j
1 OFFLINE OFFLINE
ora.prod01.vip
1 ONLINE ONLINE prod01
ora.prod02.vip
1 ONLINE ONLINE prod02
ora.scan1.vip
1 ONLINE ONLINE prod01
[[email protected] bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 96e6e83cf3364f9abf13c9310830d798 (/dev/asm-diskb) [DATA]
2. ONLINE 3d317981258b4fcfbf64f119f41a7f28 (/dev/asm-diskc) [DATA]
3. ONLINE 3aa046d1930c4f39bf9bcf3297554d85 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[[email protected] bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2816
Available space (kbytes) : 259304
ID : 671775796
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[[email protected] bin]#
asm spfile恢復
create spfile='+DATA' from memory.
[[email protected] ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:24:06 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string
SQL> show parameter asm_disk
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string
asm_diskstring string
SQL> alter system set asm_diskstring='/dev/*';
System altered.
SQL> show parameter asm_disk
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string
asm_diskstring string /dev/*
SQL> create spfile='+data' from memory;
File created.
SQL>
CRS-4602錯誤故障:
[[email protected] bin]# ./crsctl replace votedisk +DATA
CRS-4602: Failed 27 to add voting file 1d4482d87eea4f84bfb02d760f922b85.
CRS-4602: Failed 27 to add voting file 6dc3d69ccf204f28bf54faa59949aa32.
CRS-4602: Failed 27 to add voting file 5c44d63828734fbebf1cf9c70052cab6.
Failed to replace voting disk group with +DATA.
CRS-4000: Command Replace failed, or completed with errors.
[[email protected] bin]#
由於asm spfile檔案丟失導致asm_diskstring引數為空,導致該問題:
[[email protected] prod01]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:13:18 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter system set asm_diskstring='/dev/*';
System altered.
SQL>