1. 程式人生 > >oracle grid ocr磁碟組丟失恢復

oracle grid ocr磁碟組丟失恢復

版本:11.2.0.4.0
Reference : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) (Doc ID 428681.1)

CRS-4256 CRS-4602 While Replacing Voting Disk (文件 ID 1475588.1)
How to Restore ASM Based OCR After Complete Loss of the CRS Diskgroup on Linux/Unix Systems (文件 ID 1062983.1)

環境檢查:

確認ocr,votedisk 磁碟位置。

[[email protected] ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   309717e185584f38bf0380d7e53e95ee (/dev/asm-diskb) [DATA]
 2. ONLINE   791a04adb2534f28bf4954cfa12289a5 (/dev/asm-diskc) [DATA]
 3. ONLINE   af884b0a019d4f77bf3c0269106a3c95 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[
[email protected]
~]$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2828 Available space (kbytes) : 259292 ID : 671775796 Device/File Name : +DATA Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user

模擬破壞磁碟

[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskb bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.72446 s, 608 MB/s
[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskc bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.75642 s, 597 MB/s
[[email protected] gpnpd]# dd if=/dev/zero of=/dev/asm-diskd bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.91177 s, 548 MB/s
[[email protected] gpnpd]# 

檢查叢集狀態

[[email protected] ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

重建磁碟組,

在一個節點操作,其它節點處於關閉狀態。
/u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
建立data磁碟組,修改 asm_diskstring引數。

[[email protected] ~]# /u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'prod01'
CRS-2676: Start of 'ora.mdnsd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prod01'
CRS-2676: Start of 'ora.gpnpd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prod01'
CRS-2672: Attempting to start 'ora.gipcd' on 'prod01'
CRS-2676: Start of 'ora.cssdmonitor' on 'prod01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prod01'
CRS-2672: Attempting to start 'ora.diskmon' on 'prod01'
CRS-2676: Start of 'ora.diskmon' on 'prod01' succeeded
CRS-2676: Start of 'ora.cssd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'prod01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2672: Attempting to start 'ora.ctssd' on 'prod01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2676: Start of 'ora.drivers.acfs' on 'prod01' succeeded
CRS-2676: Start of 'ora.ctssd' on 'prod01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prod01'
CRS-2676: Start of 'ora.asm' on 'prod01' succeeded

[[email protected] ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 13:59:13 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create diskgroup DATA normal redundancy
DISK  '/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd'
attribute 'compatible.asm'='11.2';  2    3  
Diskgroup created.

SQL>  alter system set asm_diskstring='/dev/*';

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[[email protected] ~]$ 

恢復ocr,votedisk

ocrconfig -showbackup

root:

/u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr

grid

[[email protected] ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

prod02     2018/11/21 09:00:45     /u01/app/11.2.0/grid/cdata/prod-cluster/backup00.ocr

prod02     2018/11/21 05:00:45     /u01/app/11.2.0/grid/cdata/prod-cluster/backup01.ocr

prod02     2018/11/21 01:00:44     /u01/app/11.2.0/grid/cdata/prod-cluster/backup02.ocr

prod02     2018/11/20 01:00:42     /u01/app/11.2.0/grid/cdata/prod-cluster/day.ocr

prod02     2018/11/12 21:00:25     /u01/app/11.2.0/grid/cdata/prod-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

#root 恢復ocr
[[email protected] ~]# /u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr 

#root 恢復votedisk
[[email protected] bin]# ./crsctl replace votedisk +DATA
Successful addition of voting disk 96e6e83cf3364f9abf13c9310830d798.
Successful addition of voting disk 3d317981258b4fcfbf64f119f41a7f28.
Successful addition of voting disk 3aa046d1930c4f39bf9bcf3297554d85.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced

驗證叢集狀態

關閉叢集# crsctl stop crs -f
啟動所有節點叢集:# crsctl start crs

[[email protected] bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.DG1.dg
               ONLINE  OFFLINE      prod01                                       
               ONLINE  OFFLINE      prod02                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.asm
               ONLINE  ONLINE       prod01                   Started             
               ONLINE  ONLINE       prod02                   Started             
ora.gsd
               OFFLINE OFFLINE      prod01                                       
               OFFLINE OFFLINE      prod02                                       
ora.net1.network
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.ons
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.registry.acfs
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod01                                       
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.oc4j
      1        OFFLINE OFFLINE                                                   
ora.prod01.vip
      1        ONLINE  ONLINE       prod01                                       
ora.prod02.vip
      1        ONLINE  ONLINE       prod02                                       
ora.scan1.vip
      1        ONLINE  ONLINE       prod01                                       
[[email protected] bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   96e6e83cf3364f9abf13c9310830d798 (/dev/asm-diskb) [DATA]
 2. ONLINE   3d317981258b4fcfbf64f119f41a7f28 (/dev/asm-diskc) [DATA]
 3. ONLINE   3aa046d1930c4f39bf9bcf3297554d85 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[[email protected] bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       2816
     Available space (kbytes) :     259304
     ID                       :  671775796
     Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

     Cluster registry integrity check succeeded

     Logical corruption check succeeded

[[email protected] bin]#

asm spfile恢復

create spfile='+DATA' from memory.

[[email protected] ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:24:06 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> show parameter spfile;

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
spfile                     string
SQL> show parameter asm_disk             

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                 string
asm_diskstring                 string
SQL> alter system set asm_diskstring='/dev/*';

System altered.

SQL> show parameter asm_disk  

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                 string
asm_diskstring                 string     /dev/*
SQL> create spfile='+data' from memory;

File created.

SQL>

CRS-4602錯誤故障:

[[email protected] bin]# ./crsctl replace votedisk +DATA
CRS-4602: Failed 27 to add voting file 1d4482d87eea4f84bfb02d760f922b85.
CRS-4602: Failed 27 to add voting file 6dc3d69ccf204f28bf54faa59949aa32.
CRS-4602: Failed 27 to add voting file 5c44d63828734fbebf1cf9c70052cab6.
Failed to replace voting disk group with +DATA.
CRS-4000: Command Replace failed, or completed with errors.
[[email protected] bin]#

由於asm spfile檔案丟失導致asm_diskstring引數為空,導致該問題:

[[email protected] prod01]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:13:18 2018

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter system set asm_diskstring='/dev/*';

System altered.

SQL>