Oracle 11gR2 RAC恢復OCR和VOTE DISK
Oracle 11gR2 RAC恢復OCR和VOTE DISK
原文連結：
http://www.askmaclean.com/archives/11-2-lost-ocr-votedisk-group-recovery.html
之前有同學在我的Oracle Allstarts群裡討論關於丟失包含ocr和votedisk的ASM diskgroup導致11gR2 RAC cluster無法正常啟動的問題，最早我在《在11gR2 RAC中修改ASM DISK Path磁碟路徑》一文中介紹了，如何在不啟動CRS的情況下啟動11.2中的ASM例項並實施操作， 這裡也需要用到同樣的命令”crsctl start crs -excl -nocrs “;
注意以下的恢復手段，針對ASM中單獨的ocr或者單獨的votedisk丟失也有效，因為11.2中普遍把ocr和votedisk存放在ASM中，而ASM的啟動又依賴於ocr和votedisk，所以在丟失ocr或votedisk仍一一者都會導致cluter無法正常啟動；這裡我們僅僅討論如何讓CRS正常啟動，如果丟失的diskgroup中還存放有資料庫的話，資料的恢復不屬於本篇文章的討論範疇。
前提：恢復的前提是你仍有和故障前一樣多的ASM LUN DISK，且你有OCR的自動備份，注意預設情況下每4個小時會自動備份一次，只要你沒有刪除$GI_HOME，一般都會有備份可用；不要求有votedisk備份
恢復場景： 利用dd命令清空ocr和votedisk所在diskgroup header，模擬diskgroup corruption:
1. 檢查votedisk和 ocr備份
[[email protected] ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group — —— ———————— ———— ——— 1. ONLINE a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG] 2. ONLINE a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG] 3. ONLINE 36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG] 4. ONLINE af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG] 5. ONLINE 3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG] Located 5 voting disk(s).
[[email protected] ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy vrh1 2012/08/09 01:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup00.ocr vrh1 2012/08/08 21:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup01.ocr vrh1 2012/08/08 17:59:55 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup02.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/day.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/week.ocr PROT-25: Manual backups for the Oracle Cluster Registry are not available
2. 徹底關閉所有節點上的clusterware ,OHASD
[[email protected] ~]$crsctl stop has -f
3. 請養成良好的習慣，做危險操作前備份asm header
4. 使用dd 命令 破壞ocr和votedisk所在diskgroup
[[email protected] ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00423853 seconds, 247 MB/s [[email protected] ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0045179 seconds, 232 MB/s [[email protected] ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00469976 seconds, 223 MB/s [[email protected] ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00344262 seconds, 305 MB/s [[email protected] ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053518 seconds, 196 MB/s
5. 在一個節點上嘗試重新啟動HAS
[[email protected] ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
但是因為ocr和votedisk所在diskgroup丟失，所以CSS將無法正常啟動，如以下日誌所示：
檢視日誌：alertvrh1.log
-
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log -
2012-08-09 03:35:41.207 -
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log -
2012-08-09 03:35:56.240 -
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log -
2012-08-09 03:36:11.284 -
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log -
2012-08-09 03:36:26.305 -
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log -
2012-08-09 03:36:41.328
ocssd.log：
2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*) 2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2012-08-09 03:40:26.662: [ SKGFD][1078700352]Discovery with str:/dev/asm*: 2012-08-09 03:40:26.662: [ SKGFD][1078700352]UFS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskb: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskj: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskh: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskc: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskd: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diske: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskg: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diski: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskk: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]OSS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xdf22a0 from lib :UFS:: for disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xf412a0 from lib :UFS:: for disk :/dev/asm-diskb: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf3a680 from lib :UFS:: for disk :/dev/asm-diskj: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf93da0 from lib :UFS:: for disk :/dev/asm-diskh: 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvDiskVerify: Successful discovery of 0 disks 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvFindInitialConfigs: No voting files found 2012-08-09 03:40:26.667: [ CSSD][1078700352](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
正式的恢復ocr和votedisk所在diskgroup的步驟如下：
1. 以-excl -nocrs 方式啟動cluster，這將可以啟動ASM例項 但不啟動CRS
[[email protected] vrh1]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘vrh1’ CRS-2676: Start of ‘ora.mdnsd’ on ‘vrh1’ succeeded CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘vrh1’ CRS-2676: Start of ‘ora.gpnpd’ on ‘vrh1’ succeeded CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘vrh1’ CRS-2672: Attempting to start ‘ora.gipcd’ on ‘vrh1’ CRS-2676: Start of ‘ora.cssdmonitor’ on ‘vrh1’ succeeded CRS-2676: Start of ‘ora.gipcd’ on ‘vrh1’ succeeded CRS-2672: Attempting to start ‘ora.cssd’ on ‘vrh1’ CRS-2672: Attempting to start ‘ora.diskmon’ on ‘vrh1’ CRS-2676: Start of ‘ora.diskmon’ on ‘vrh1’ succeeded CRS-2676: Start of ‘ora.cssd’ on ‘vrh1’ succeeded CRS-2679: Attempting to clean ‘ora.cluster_interconnect.haip’ on ‘vrh1’ CRS-2672: Attempting to start ‘ora.ctssd’ on ‘vrh1’ CRS-2681: Clean of ‘ora.cluster_interconnect.haip’ on ‘vrh1’ succeeded CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘vrh1’ CRS-2676: Start of ‘ora.ctssd’ on ‘vrh1’ succeeded CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘vrh1’ succeeded CRS-2672: Attempting to start ‘ora.asm’ on ‘vrh1’ CRS-2676: Start of ‘ora.asm’ on ‘vrh1’ succeeded
2.重建原ocr和votedisk所在diskgroup,注意compatible.asm必須是11.2:
[[email protected] vrh1]# su - grid [[email protected] ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 04:16:58 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup systemdg high redundancy disk ‘/dev/asm-diskh’,’/dev/asm-diskg’,’/dev/asm-diski’,’/dev/asm-diskj’,’/dev/asm-diskk’ ATTRIBUTE ‘compatible.rdbms’ = ’11.2’, ‘compatible.asm’ = ’11.2’;
3.從ocr backup中恢復ocr並做ocrcheck檢驗:
[[email protected] ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr
[[email protected] ~]# ocrcheck
Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3180 Available space (kbytes) : 258940 ID : 1238458014 Device/File Name : +systemdg Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded
4. 準備恢復votedisk ，可能會遇到下面的錯誤:
[[email protected] ~]$ crsctl replace votedisk +SYSTEMDG
CRS-4602: Failed 27 to add voting file 2e4e0fe285924f86bf5473d00dcc0388. CRS-4602: Failed 27 to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389. CRS-4602: Failed 27 to add voting file a109ead9ea4e4f28bfe233188623616a. CRS-4602: Failed 27 to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3. CRS-4602: Failed 27 to add voting file 7b5a8cd24f954fafbf835ad78615763f. Failed to replace voting disk group with +SYSTEMDG. CRS-4000: Command Replace failed, or completed with errors.
需要重新配置一下ASM的引數，並重啟ASM:
SQL> alter system set asm_diskstring=‘/dev/asm*’; System altered. SQL> create spfile from memory; File created. SQL> startup force mount; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE —————————————————— ————— ——————————————— spfile string /g01/11.2.0/grid/dbs/spfi le+ASM1.ora [[email protected] trace]$ crsctl replace votedisk +SYSTEMDG CRS-4256: Updating the profile Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a. Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840. Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce. Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5. Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f. Successfully replaced voting disk group with +SYSTEMDG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced
5. 重啟has服務，檢驗cluster是否正常：
[[email protected] ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online
[[email protected] ~]# crsctl query css votedisk
-
## STATE File Universal Id File Name Disk group -
——————————————— —— ———————— ———— ———— -
1. ONLINE 85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG] -
2. ONLINE 201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG] -
3. ONLINE 6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG] -
4. ONLINE 93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG] -
5. ONLINE 3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG] -
Located 5 voting disk(s).
[[email protected] ~]# crsctl stat res -t
———————————————————————————————————————— NAME TARGET STATE SERVER STATE_DETAILS Local Resources ———————————————————————————————————————— ora.BACKUPDG.dg ONLINE ONLINE vrh1 ora.DATA.dg ONLINE ONLINE vrh1 ora.LISTENER.lsnr ONLINE ONLINE vrh1 ora.LSN_MACLEAN.lsnr ONLINE ONLINE vrh1 ora.SYSTEMDG.dg ONLINE ONLINE vrh1 ora.asm ONLINE ONLINE vrh1 ......