private 網絡不穩定引起的Evicting instance 2 from cluster
環境：雙節點RAC, oracle 11.2.3
客戶電話RAC實例２異常，現場查看日誌：
實例２：
Fri Aug 25 09:45:16 2017
Received an instance abort message from instance 1Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.Please check instance 1 alert and LMON trace files for detail.
LMS0 (ospid: 24510820): terminating the instance due to error 481
Fri Aug 25 09:45:16 2017
System state dump requested by (instance=2, osid=24510820 (LMS0)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/11.2.0/diag/rdbms/ins/ins2/trace/ins2_diag_21561818.trc
Instance terminated by LMS0, pid = 24510820
實例１
Fri Aug 25 09:44:25 2017
IPC Send timeout detected. Sender: ospid 35783054 [[email protected] (LMS1)]
Receiver: inst 2 binc 2073329022 ospid 24183072
IPC Send timeout to 2.2 inc 28 for msg type 65518 from opid 14
Fri Aug 25 09:44:27 2017
Communications reconfiguration: instance_number 2
Fri Aug 25 09:45:16 2017
Detected an inconsistent instance membership by instance 1
Evicting instance 2 from cluster
Waiting for instances to leave: 2
Fri Aug 25 09:45:16 2017
Dumping diagnostic data in directory=[cdmp_20170825094516], requested by (instance=2, osid=24510820 (LMS0)), summary=[abnormal instance termination].
Reconfiguration started (old inc 28, new inc 32)
List of instances:
1 (myinst: 1)
查看/oracle/11.2.0/diag/rdbms/gjj/ins2/trace/ins2_diag_21561818.trc
*** 2017-08-25 14:24:35.900
I‘m the voting node
Group reconfiguration cleanup
confirm->incar_num 22, rcfgctx->prop_incar 0
Send my bitmap to master 0
kjzgmappropose : incar 0, newmap -
3000000000000000000000000000000000000000000000000000000000000000
kjzgmappropose : rc from psnd : 30
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
懷疑心跳網絡存在問題（這套RAC之前就出現過幾次實例被驅逐的問題，但實例自動都啟動了，這次實例被驅逐後實例2不能啟動，針對之前實例被驅逐的問題進行過參數修改，通過這次的情況來看，實該不是參數設置的問題）。
測試心跳網絡，連通性和傳輸速率都沒有問題，後續打算通過haip進一步提升心跳網絡可用性，在添加haip過程中發現當服和服務器和交換機新添加網絡後出來數據包丟失的情況，丟包率50%，判斷心跳網絡穩定性存在問題，基於此撤掉新添加的心跳線，更換原來的心跳線，重啟被驅逐的實例2，實例正常。
最後判斷是原心跳線RJ45頭存在某兩芯短路的問題造成此次故障。
本文出自 “技術門徒” 博客，請務必保留此出處http://lizhhui.blog.51cto.com/3222144/1962757
private 網絡不穩定引起的Evicting instance 2 from cluster