Oracle 11gR2 下RAC 安裝後，啟動CRS. 錯誤如下：

[[email protected] bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

從這個錯誤提示，可以看到是CRS啟動失敗了。 CRS是關鍵程序。 它不能啟動，Clusterware 也是啟動不了。 導致這個問題的原因很多。

Log 如下：

[[email protected] rac1]# tail -50 /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log

ORA-15077: could not locate ASM instance serving a required diskgroup

2010-11-16 17:13:44.286: [  OCRASM][3046411024]proprasmo: kgfoCheckMount returned [7]

2010-11-16 17:13:44.286: [  OCRASM][3046411024]proprasmo: The ASM instance is down

2010-11-16 17:13:44.287: [  OCRRAW][3046411024]proprioo: Failed to open [+CRS]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.

2010-11-16 17:13:44.287: [  OCRRAW][3046411024]proprioo: No OCR/OLR devices are usable

2010-11-16 17:13:44.287: [  OCRASM][3046411024]proprasmcl: asmhandle is NULL

2010-11-16 17:13:44.287: [  OCRRAW][3046411024

2010-11-16 17:13:44.287: [  OCRASM][3046411024]proprasmcl: asmhandle is NULL

2010-11-16 17:13:44.287: [  OCRAPI][3046411024]a_init:16!: Backend init unsuccessful : [26]

2010-11-16 17:13:44.288: [  CRSOCR][3046411024] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

] [7]

2010-11-16 17:13:44.288: [    CRSD][3046411024][PANIC] CRSD exiting: Could not init OCR, code: 26

2010-11-16 17:13:44.288: [    CRSD][3046411024] Done.

       這裡的提示是ASM 沒有啟動造成的。 這裡牽涉到的問題較複雜。

       這篇文章不打算去具體分析這個問題。 Oracle 官網上有一篇文章對這個問題進行了非常詳細的說明。轉到了我的Blog。 參考：

       How to Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]

In this Document

  Goal

  Solution

     Start up sequence:

     Cluster status

     Case 1: OHASD.BIN does not start

     Case 2: OHASD Agents does not start

     Case 3: CSSD.BIN does not start

     Case 4: CRSD.BIN does not start

     Case 5: GPNPD.BIN does not start

     Case 6: Various other daemons does not start

     Case 7: CRSD Agents does not start

     Network and Naming Resolution Verification

     Log File Location, Ownership and Permission

     Network Socket File Location, Ownership and Permission

     Diagnostic file collection

  References

在這裡寫下我分析問題的思路：

1. 根據log，看能否找到問題的原因。 如果不能清楚的定位問題。 就只能繼續分析。

2. 根據CRS 啟動的順序來分析。

       在啟動的時候，要先啟動ASM 例項， 這裡牽涉到儲存問題。

       （1）網路是否正常

       （2）儲存是否正常的對映到相關的位置， 我的實驗採用的是multipath，將儲存對映到/dev/mapper/* 目錄下。 在遇到問題的時候，會去檢查這個問題是否有相關的對映。

       （3）儲存的許可權問題。 因為對映之後，預設是的root使用者。 我在/etc/rc.d/rc.local 檔案裡添加了改變許可權的指令碼。 開機啟動的時候，就將相關對映檔案改成Oracle 使用者。

3.  如果這些都正常，沒有問題， 可以嘗試重啟CRS 或者重啟作業系統。

補充：

       在網上還搜尋到一個導致CSSD啟動失敗的原因。 這個我關注的是，它講到了一個知識點。 講到了 /tmp/.oracle 和 /var/tmp/.oracle 這兩個目錄的作用。每次Server重啟的時候，會在這兩個檔案裡存放鎖的資訊。 當某次重啟後，這兩個檔案不能被刪除，就會導致鎖不能更新，從而不能啟動。

由此也理解了，在刪除Clusterware的時候，為什麼需要刪除這2個目錄了。

在RAC 刪除的那篇文件裡提到了解除安裝RAC時要刪除這2個目錄。 參考：

       RAC 解除安裝說明

crs.log 日誌內容：

2007-04-11 14:37:34.020: [ COMMCRS][1693]clsc_connect: (100f78610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_drdb1_crs))

2007-04-11 14:37:34.020: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

2007-04-11 14:37:34.021: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

2007-04-11 14:37:35.740: [ COMMCRS][1695]clsc_connect: (100f78610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_drdb1_crs))

2007-04-11 14:37:35.740: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

When we checked ocssd.log it contained the following

CSSD]2007-04-11 12:53:56.211 [6] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//dev/rdsk/c5t8d0s5)
[ CSSD]2007-04-11 12:53:56.211 [10] >TRACE: clssnmvKillBlockThread: spawned for disk 1 (/dev/rdsk/c5t9d0s5) initial sleep interval (1000)ms
[ CSSD]2007-04-11 12:53:56.211 [11] >TRACE: clssnmvKillBlockThread: spawned for disk 0 (/dev/rdsk/c5t8d0s5) initial sleep interval (1000)ms
[ CSSD]2007-04-11 12:53:56.228 [1] >TRACE: clssnmFatalInit: fatal mode enabled
[ CSSD]2007-04-11 12:53:56.269 [13] >TRACE: clssnmconnect: connecting to node 1, flags 0×0001, connector 1
[ CSSD]2007-04-11 12:53:56.274 [13] >TRACE: clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=drdb1-priv)(PORT=49895))

[ CSSD]2007-04-11 12:53:56.274 [13] >TRACE: clssnmconnect: connecting to node 0, flags 0×0000, connector 1
[ CSSD]2007-04-11 12:53:56.279 [14] >TRACE: clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))

[ CSSD]2007-04-11 12:53:56.279 [14] >ERROR: clssgmclientlsnr: listening failed for (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1)) (3)
[ CSSD]2007-04-11 12:53:56.279 [14] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))
[ CSSD]2007-04-11 12:53:56.279 [14] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_drdb1_crs))
[ CSSD]2007-04-11 13:07:36.516 >USER: Oracle Database 10g CSS Release 10.2.0.2.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
[ clsdmt]Fail to listen to (ADDRESS=(PROTOCOL=ipc)(KEY=drdb1DBG_CSSD))
[ CSSD]2007-04-11 13:07:36.516 >USER: CSS daemon log for node drdb1, number 1, in cluster crs
[ clsdmt]Terminating clsdm listening thread
[ CSSD]2007-04-11 13:07:36.536 [1] >TRACE: clssscmain: local-only set to false
[ CSSD]2007-04-11 13:07:36.545 [1] >TRACE: clssnmReadNodeInfo: added node 1 (drdb1) to cluster
[ CSSD]2007-04-11 13:07:36.588 [5] >TRACE: clssnm_skgxnmon: skgxn init failed, rc 1
[ CSSD]2007-04-11 13:07:36.588 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor

解決方法：

       By checking the above logs we have realised the listener of CSS deamon was unable to start.

       the reason why it was unable to start was that each time server reboots it creates a socket at /tmp/.oracle or /var/tmp/.oracle directory .

       Also if there are previously existing sockets they cannot be reused or deleted automatically from this directory .oracle.

       Therefore the solution to above problem was obtained by deleting all the files inside .oracle directoery in /var/tmp or /tmp.

Hence the crs started and cluster came up.

2

解決：

mv  /var/log/.oracle  /var/log/.oracle.bak