一個dg警告發現的硬體問題 (r6筆記第60天)
今天收到一條報警簡訊，提示dg似乎出了點問題。資訊的來源是從v$dataguard_status裡面掃描得到的最新錯誤。
2015-09-15 22:06:19.0 Log Transport ServicesErrorError 12541 received logging on to the standby
2015-09-15 22:06:19.0 Log Transport ServicesErrorPING[ARC1]: Heartbeat failed to connect to standby 'stest11g'. Error is 12541.
看樣子是心跳的檢測失敗了，看來主庫和備庫之間的網路可能出現了延遲之類的問題，在最大效能模式下，這個還是能夠接受的，當時就沒有在意。
等過了一會之後，自己想還是看看到底是怎麼回事吧，畢竟別的庫就沒有保護這樣的錯誤。
連線到備庫之後，發現ssh連線是正常的，證明備庫還是能夠訪問的，沒有出現備庫的宕機問題。
檢視例項也存在，但是監聽器給停掉了。自己也感覺挺奇怪，監聽怎麼會自動停掉呢。就手工啟動，結果啟動就報了下面的錯誤。
lsnrctl start listener
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 14-SEP-2015 23:27:38
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Starting /DATA/app/oracle/product/11.2.0.4/bin/tnslsnr: please wait...
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
$ ll log
ls: reading directory log: Input/output error
total 0
ls: reading directory .: Input/output error
這個錯誤又陌生了，檢視資料顯示應該是檔案系統出了問題。
怎麼 驗證呢，使用dmesg來做或者使用MegaCli都可以。
> dmesg|grep sd|less
sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
sd 0:2:0:0: [sda] CDB: Write(10): 2a 00 13 f4 21 38 00 00 08 00
Buffer I/O error on device sda8, logical block 33423143
lost page write due to I/O error on sda8
end_request: I/O error, dev sda, sector 331257944
Buffer I/O error on device sda8, logical block 32984587
lost page write due to I/O error on sda8
end_request: I/O error, dev sda, sector 303472624
end_request: I/O error, dev sda, sector 628396920
JBD2: Detected IO errors while flushing file data on sda8-8
Aborting journal on device sda8-8.
EXT4-fs error (device sda8) in add_dirent_to_buf: Journal has aborted
EXT4-fs error (device sda8): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (sda8): Remounting filesystem read-only
EXT4-fs error (device sda8) in ext4_reserve_inode_write: Journal has aborted
EXT4-fs error (device sda8) in ext4_reserve_inode_write: Journal has aborted
EXT4-fs error (device sda8) in ext4_reserve_inode_write: Journal has aborted
EXT4-fs error (device sda8) in ext4_reserve_inode_write: Journal has aborted