【11grac】Oracle RAC 更換儲存實驗
【11grac】Oracle RAC 更換儲存實驗
回到未來望過去2019-05-14 10:49:45478收藏2 分類專欄:rac rac專欄收錄該內容 27 篇文章0 訂閱 訂閱專欄實驗環境準備:
RHEL 6.5 + Oracle 11.2.0.4 RAC (2nodes)
- OCR和Voting Disk使用的是OCR1磁碟組,底層對應3個1G大小的共享LUN,一般冗餘;
- DATA使用的是DATA1磁碟組,底層對應的是3個5G大小的共享LUN,外部冗餘;
- FRA使用的是FRA1磁碟組,底層對應的是1個5G大小的共享LUN,外部冗餘;
- 額外新增7個共享LUN(3個1G大小,4個5G大小),用來模擬新的儲存映射出來的共享LUN;
我這裡是使用Oracle VM VirtualBox 4.3虛擬機器軟體模擬這個環境,
預設初始的RAC環境已經安裝完畢,且已經存有業務資料。
對於RAC環境的安裝可參考:
- Linux平臺 Oracle 11gR2 RAC安裝Part1:準備工作
- Linux平臺 Oracle 11gR2 RAC安裝Part2:GI安裝
- Linux平臺 Oracle 11gR2 RAC安裝Part3:DB安裝
模擬的業務資料可參考模擬業務最小測試用例01第1~5步進行初始化。
初始的RAC環境配置資訊:
- 192.168.56.150 jyrac1
-
192.168.56.152 jyrac2
- 10.10.10.11 jyrac1-priv
- 10.10.10.12 jyrac2-priv
- 192.168.56.151 jyrac1-vip
- 192.168.56.153 jyrac2-vip
- 192.168.56.160 jyrac-scan
本次需求:
更換RAC的共享儲存,即把之前的儲存裝置劃出的共享LUN上的所有資料全部遷移新的儲存裝置劃分的LUN上。
具體就涉及到OCR,voting disk遷移,DATA遷移。
在我這裡實驗具體就是指OCR1,DATA1,FRA1磁碟組的遷移。
一、準備工作
二、更換儲存
一、準備工作
1.1檢視asm磁碟組
select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
- SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup;
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
1.2檢視asm的磁碟資訊
select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2;
- SQL> set linesize 1000
- SQL> r
- 1* select GROUP_NUMBER, DISK_NUMBER, TOTAL_MB, FREE_MB, NAME, PATH from v$asm_disk order by 1,2
- GROUP_NUMBER DISK_NUMBER TOTAL_MB FREE_MB NAME PATH
- 1 0 1024 714 OCR1_0000 /dev/asm-diskb
- 1 1 1024 716 OCR1_0001 /dev/asm-diskc
- 1 2 1024 716 OCR1_0002 /dev/asm-diskd
- 2 0 5120 4552 DATA1_0000 /dev/asm-diske
- 2 1 5120 4549 DATA1_0001 /dev/asm-diskf
- 2 2 5120 4549 DATA1_0002 /dev/asm-diskg
- 3 0 5120 4790 FRA1_0000 /dev/asm-diskh
- 7 rows selected.
1.3檢視儲存裝置資訊,確定新的儲存
這裡實驗環境是使用virtualbox模擬新增7個大小和之前一樣的共享儲存;
即分別為 1G大小的3個(OCR2),5G大小的3個(DATA1),5G大小的1個(FRA1)。
NewStorage1
NewStorage2
NewStorage3
NewStorage4
NewStorage5
NewStorage6
NewStorage7
在系統中就是/dev/sdi j k l m n o
udev繫結(兩個節點root執行)
- --New Shared Disks
- for i in i j k l m n o;
- do
- echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
- done
udev繫結後對應就是asm-diski,j,k,l,m,n,o
start_udev (兩個節點root執行)
檢視某個磁碟大小:
- [root@jyrac1 ~]# fdisk -l /dev/asm-diski
- Disk /dev/asm-diski: 1073 MB, 1073741824 bytes
- 255 heads, 63 sectors/track, 130 cylinders
- Units = cylinders of 16065 * 512 = 8225280 bytes
- Sector size (logical/physical): 512 bytes / 512 bytes
- I/O size (minimum/optimal): 512 bytes / 512 bytes
- Disk identifier: 0x00000000
確定/dev/asm-diski,/dev/asm-diskj,/dev/asm-diskk對應的是新加的1G大小的3個盤
用於建立新的OCR2
二、更換儲存
2.1遷移OCR1到OCR2
2.1.1 建立新的OCR2磁碟組
su – grid
sqlplus / as sysasm
CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK '/dev/asm-diski', '/dev/asm-diskj', '/dev/asm-diskk' ATTRIBUTE 'compatible.asm'='11.2';
- SQL> CREATE DISKGROUP OCR2 NORMAL REDUNDANCY DISK '/dev/asm-diski', '/dev/asm-diskj', '/dev/asm-diskk' ATTRIBUTE 'compatible.asm'='11.2';
- Diskgroup created.
檢視兩個節點是否添加了OCR2:
- --節點1:
- SQL> r
- 1* select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- ------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
- OCR2 3072 2913 MOUNTED NORMAL 11.2.0.0.0
- --節點2:
- SQL> select name, TOTAL_MB, FREE_MB, state, type, COMPATIBILITY from v$asm_diskgroup
- 2 ;
- NAME TOTAL_MB FREE_MB STATE TYPE COMPATIBILITY
- ------------------------------------------------------------ ---------- ---------- ---------------------- ------------ ------------------------------------------------------------------------------------------------------------------------
- DATA1 15360 13502 MOUNTED EXTERN 11.2.0.0.0
- FRA1 5120 4790 MOUNTED EXTERN 11.2.0.0.0
- OCR1 3072 2146 MOUNTED NORMAL 11.2.0.0.0
- OCR2 0 0 DISMOUNTED 0.0.0.0.0
發現節點2需要手動mount新加的ASM磁碟組
sqlplus / as sysasm
- SQL> alter diskgroup OCR2 mount;
- Diskgroup altered.
2.1.2 新增OCR資訊到OCR2
more /etc/oracle/ocr.loc
ocrconfig -add +OCR2
ocrcheck -config
more /etc/oracle/ocr.loc
- [root@jyrac1 ~]# more /etc/oracle/ocr.loc
- ocrconfig_loc=+OCR1
- local_only=FALSE
- [root@jyrac1 ~]# ocrconfig -add +OCR2
- [root@jyrac1 ~]#
- [root@jyrac1 ~]# ocrcheck -config
- Oracle Cluster Registry configuration is :
- Device/File Name : +OCR1
- Device/File Name : +OCR2
- [root@jyrac1 ~]# more /etc/oracle/ocr.loc
- #Device/file getting replaced by device +OCR2
- ocrconfig_loc=+OCR1
- ocrmirrorconfig_loc=+OCR2
- local_only=false
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE ab583d47c34a4f0ebfd46c8b04d7332b (/dev/asm-diskb) [OCR1]
- 2. ONLINE 853e95652d994fe3bfdf21ca23e65dfc (/dev/asm-diskc) [OCR1]
- 3. ONLINE 9f712a096a7b4f9bbfd811bff6462157 (/dev/asm-diskd) [OCR1]
- Located 3 voting disk(s).
2.1.3遷移VOTE DISK到新建立的ASM磁碟組OCR2上
crsctl replace votedisk +OCR2 (一個節點root執行)
先查詢:
- SQL> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
- GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
- ------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
- 1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
- 2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
- 3 OCR1 MOUNTED NORMAL 3072 2146 Y 11.2.0.0.0
- 4 OCR2 MOUNTED NORMAL 3072 2244 N 11.2.0.0.0
替換為新新增的OCR2:
- [root@jyrac1 ~]# crsctl replace votedisk +OCR2
- Successful addition of voting disk 83f3dea9f4c24fccbf05eaa163c5c087.
- Successful addition of voting disk ad21c1f0452e4f4abff906127ff9fcc3.
- Successful addition of voting disk 2771819c65984f85bf366e74ee85781a.
- Successful deletion of voting disk ab583d47c34a4f0ebfd46c8b04d7332b.
- Successful deletion of voting disk 853e95652d994fe3bfdf21ca23e65dfc.
- Successful deletion of voting disk 9f712a096a7b4f9bbfd811bff6462157.
- Successfully replaced voting disk group with +OCR2.
- CRS-4266: Voting file(s) successfully replaced
再查詢:
- SQL> r
- 1* select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup
- GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB VO COMPATIBILITY
- ------------ ------------------------------------------------------------ ---------------------- ------------ ---------- ---------- -- ------------------------------------------------------------------------------------------------------------------------
- 1 DATA1 MOUNTED EXTERN 15360 13502 N 11.2.0.0.0
- 2 FRA1 MOUNTED EXTERN 5120 4790 N 11.2.0.0.0
- 3 OCR1 MOUNTED NORMAL 3072 2242 N 11.2.0.0.0
- 4 OCR2 MOUNTED NORMAL 3072 2148 Y 11.2.0.0.0
發現已經voting file已經在OCR2上,然後在兩個節點root查詢:
ocrcheck && crsctl query css votedisk
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
- [root@jyrac2 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR1
- Device/File integrity check succeeded
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
2.1.4 建立ASM例項spfile到新建立的OCR2 ASM磁碟組上
一個節點grid使用者登入ASM例項執行:
create pfile='/tmp/asmpfile.ora' from spfile;
create spfile='+OCR2' from pfile='/tmp/asmpfile.ora';
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR1/jyrac-scan/asmparameterf
- ile/registry.253.919993739
- SQL> create pfile='/tmp/asmpfile.ora' from spfile;
- File created.
- SQL> create spfile='+OCR2' from pfile='/tmp/asmpfile.ora';
- File created.
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR1/jyrac-scan/asmparameterf
- ile/registry.253.919993739
2.1.5 刪除OCR原位置ASM磁碟組OCR1
一個節點root執行:
[root@jyrac1 ~]# ocrconfig -delete +OCR1
兩個節點root執行:
檢視OCR和VOTE新狀態與位置
[root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- [root@jyrac1 ~]# ocrconfig -delete +OCR1
- [root@jyrac1 ~]# ocrcheck && crsctl query css votedisk
- Status of Oracle Cluster Registry is as follows :
- Version : 3
- Total space (kbytes) : 262120
- Used space (kbytes) : 3116
- Available space (kbytes) : 259004
- ID : 1718040627
- Device/File Name : +OCR2
- Device/File integrity check succeeded
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Device/File not configured
- Cluster registry integrity check succeeded
- Logical corruption check succeeded
- ## STATE File Universal Id File Name Disk group
- -- ----- ----------------- --------- ---------
- 1. ONLINE 83f3dea9f4c24fccbf05eaa163c5c087 (/dev/asm-diski) [OCR2]
- 2. ONLINE ad21c1f0452e4f4abff906127ff9fcc3 (/dev/asm-diskj) [OCR2]
- 3. ONLINE 2771819c65984f85bf366e74ee85781a (/dev/asm-diskk) [OCR2]
- Located 3 voting disk(s).
- 節點2的ocrcheck輸出也沒問題,略。
2.1.6 重啟整個叢集
兩個節點root執行:
crsctl stop crs && crsctl start crs
2.1.7 將舊的磁碟組drop掉
檢查引數檔案:
- SQL> show parameter pfile
- NAME TYPE VALUE
- ------------------------------------ ---------------------- ------------------------------
- spfile string +OCR2/jyrac-scan/asmparameterf
- ile/registry.253.920008067
檢查磁碟組狀態:
- Select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
- select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
最後將舊的磁碟組mount並drop掉:
alter diskgroup OCR1 mount;
drop diskgroup OCR1 including contents;
- SQL> alter diskgroup OCR1 mount;
- Diskgroup altered.
- SQL> drop diskgroup OCR1 including contents;
- Diskgroup dropped.
注:舊的磁碟組只允許在一個節點mount,如果發現多個節點mount,需要在其他節點dismount,否則會刪除不掉舊的磁碟組。
2.2遷移DATA1
2.2.1 資料磁碟組DATA1新增
確定/dev/asm-diskl,/dev/asm-diskm,/dev/asm-diskn對應的是新加的5G大小的3個盤
用於資料磁碟組DATA1新增
alter diskgroup DATA1 add disk '/dev/asm-diskl' rebalance power 7;
alter diskgroup DATA1 add disk '/dev/asm-diskm', '/dev/asm-diskn' rebalance power 11;
- SQL> alter diskgroup DATA1 add disk '/dev/asm-diskl' rebalance power 7;
- Diskgroup altered.
- SQL> alter diskgroup DATA1 add disk '/dev/asm-diskm', '/dev/asm-diskn' rebalance power 11;
- Diskgroup altered.
這裡說明可以單個新增,也可以多個新增。
2.2.2 查詢同步狀態
- set linesize 200
- col NAME for a10
- select name,allocation_unit_size,state,type,free_mb,required_mirror_free_mb req_mi_fr_mb,usable_file_mb
- from v$asm_diskgroup;
2.2.3 刪除原儲存的LUN
所有磁碟新增進ASM磁碟組後,刪除原儲存的LUN。
查詢LUN path與asm_disk_name的對應關係
select name,path from v$asm_disk;
- SQL> select name,path from v$asm_disk
- NAME PATH
- ---------- ------------------------------------------------------------
- /dev/asm-diskc
- /dev/asm-diskd
- /dev/asm-disko
- /dev/asm-diskb
- OCR2_0001 /dev/asm-diskj
- OCR2_0000 /dev/asm-diski
- DATA1_0003 /dev/asm-diskl
- DATA1_0004 /dev/asm-diskm
- DATA1_0005 /dev/asm-diskn
- OCR2_0002 /dev/asm-diskk
- FRA1_0000 /dev/asm-diskh
- DATA1_0001 /dev/asm-diskf
- DATA1_0002 /dev/asm-diskg
- DATA1_0000 /dev/asm-diske
- 14 rows selected.
確定是DATA1_0000,DATA1_0001,DATA1_0002
刪除磁碟組DATA1中舊儲存對應的LUN:
- alter diskgroup DATA1 drop disk DATA1_0000 rebalance power 7;
- alter diskgroup DATA1 drop disk DATA1_0001,DATA1_0002 rebalance power 7;
2.2.4 確認儲存置換完成
- select disk_number,path,total_mb,free_mb from v$asm_disk;
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 2 /dev/asm-disko 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 0 /dev/asm-diskh 5120 4790
- 14 rows selected.
2.3遷移FRA1
FRA1磁碟組和DATA1磁碟組的新增一樣,
- alter diskgroup FRA1 add disk '/dev/asm-disko' rebalance power 7;
- alter diskgroup FRA1 drop disk FRA1_0000 rebalance power 7;
- select disk_number,path,total_mb,free_mb from v$asm_disk;
最後確定所有盤都置換完成:
- SQL> select disk_number,path,total_mb,free_mb from v$asm_disk;
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 1 /dev/asm-disko 5120 4943
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 0 /dev/asm-diskh 5120 4965
- 14 rows selected.
- SQL> r
- 1* select disk_number,path,total_mb,free_mb from v$asm_disk
- DISK_NUMBER PATH TOTAL_MB FREE_MB
- ----------- ------------------------------------------------------------ ---------- ----------
- 0 /dev/asm-diskc 0 0
- 1 /dev/asm-diskd 0 0
- 2 /dev/asm-diskh 0 0
- 3 /dev/asm-diske 0 0
- 4 /dev/asm-diskf 0 0
- 5 /dev/asm-diskg 0 0
- 6 /dev/asm-diskb 0 0
- 1 /dev/asm-diskj 1024 715
- 0 /dev/asm-diski 1024 717
- 1 /dev/asm-disko 5120 4790
- 3 /dev/asm-diskl 5120 4502
- 4 /dev/asm-diskm 5120 4499
- 5 /dev/asm-diskn 5120 4501
- 2 /dev/asm-diskk 1024 714
- 14 rows selected.
最終確定要替換的LUN都沒有資料了,主機側就可以操作了。
2.4測試
檢視叢集各資源狀態:
crsctl stat res -t
檢視OCR資訊:
ocrcheck && crsctl query css votedisk
查詢ASM例項的spfile檔案:
- sqlplus / as sysasm <<EOF
- show parameter pfile
- EOF
查詢之前的業務資料:
- -- 業務查詢SQL 1
- select * from t1, t2 where t1.id = t2.t1_id and t1.n = 19;
- -- 業務查詢SQL 2
- select * from t1, t2 where t1.id = t2.t1_id;
至此,整個RAC環境更換儲存的實驗就全部完成了。