ORA-00471: DBWR process terminated with error案例
每年年底，系統管理員都要組織一次容災方案的測試、演練。會在一個與生產環境網路隔離的DR環境中，啟動各個“生產環境伺服器”，然後讓各路人員參與其中測試、演練容災方案是否可靠。這次演練中，一臺Oracle資料庫伺服器啟動的時候遇到了問題。如下所示，啟動的時候遇到ORA-03113: end-of-file on communication channel錯誤。
[[email protected] ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.5.0 - Production onFri Dec 21 09:42:11 2018
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
Connected to an idle instance.
SQL> startup
ORA-03113: end-of-file on communication channel
SQL>
檢查告警日誌，發現數據庫在啟動的時候，報“ORA-00471: DBWR process terminated with error
PMON started with pid=2, OS id=25005
PSP0 started with pid=3, OS id=25007
MMAN started with pid=4, OS id=25009
DBW0 started with pid=5, OS id=25011
LGWR started with pid=6, OS id=25013
CKPT started with pid=7, OS id=25016
SMON started with pid=8, OS id=25018
RECO started with pid=9, OS id=25020
CJQ0 started with pid=10, OS id=25022
MMON started with pid=11, OS id=25024
Fri Dec 21 09:44:36 CST 2018
starting up 8 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=25026
Fri Dec 21 09:45:12 CST 2018
starting up 24 shared server(s) ...
Fri Dec 21 09:46:43 CST 2018
Errors in file /u01/app/oracle/admin/SCM2/bdump/scm2_pmon_25005.trc:
ORA-00471: DBWR process terminated with error
Fri Dec 21 09:46:43 CST 2018
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 25005
啟動資料庫例項的時候，報“ORA-00471: DBWR process terminated with error”這個錯誤，這個很蹊蹺，很有可能是程序被系統給Kill掉了，檢查作業系統的錯誤日誌，發現出現了oom_kill_process,也就是說資料庫例項啟動的時候，由於系統記憶體資源緊張，DBWR程序被系統選作了犧牲品。具體錯誤日誌如下所示：
Dec 21 09:46:39 mylnx6 kernel: oracle invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Dec 21 09:46:39 mylnx6 kernel: oracle cpuset=/ mems_allowed=0
Dec 21 09:46:39 mylnx6 kernel: Pid: 25026, comm: oracle Not tainted 2.6.32-200.13.1.el5uek #1
Dec 21 09:46:39 mylnx6 kernel: Call Trace:
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d725a>] ? find_get_page+0x22/0x69
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff8106d7cd>] ? getrusage+0x2b1/0x2ce
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff8101270e>] ? common_interrupt+0xe/0x13
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81043696>] ? should_resched+0xe/0x2f
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
Dec 21 09:46:39 mylnx6 kernel: Mem-Info:
Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA32 per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: Node 0 Normal per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
Dec 21 09:46:41 mylnx6 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
Dec 21 09:46:41 mylnx6 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: active_anon:1764 inactive_anon:209 isolated_anon:64
Dec 21 09:46:41 mylnx6 kernel: active_file:349 inactive_file:1710 isolated_file:0
Dec 21 09:46:41 mylnx6 kernel: unevictable:5377 dirty:0 writeback:4 unstable:0
Dec 21 09:46:41 mylnx6 kernel: free:29838 slab_reclaimable:2400 slab_unreclaimable:119491
Dec 21 09:46:41 mylnx6 kernel: mapped:2703 shmem:830 pagetables:9849 bounce:0
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA free:15652kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15172kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 3000 24210 24210
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32 free:86296kB min:2464kB low:3080kB high:3696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 21210 21210
Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal free:17404kB min:17440kB low:21800kB high:26160kB active_anon:7056kB inactive_anon:836kB active_file:1396kB inactive_file:6840kB unevictable:21508kB isolated(anon):256kB isolated(file):0kB present:21719040kB mlocked:21504kB dirty:0kB writeback:16kB mapped:10812kB shmem:3320kB slab_reclaimable:9600kB slab_unreclaimable:477964kB kernel_stack:2800kB pagetables:39396kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 0 0
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15652kB
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32: 12*4kB 13*8kB 2*16kB 5*32kB 5*64kB 11*128kB 3*256kB 7*512kB 6*1024kB 4*2048kB 16*4096kB = 86296kB
Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal: 420*4kB 1917*8kB 49*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17800kB
Dec 21 09:46:41 mylnx6 kernel: 4722 total pagecache pages
Dec 21 09:46:41 mylnx6 kernel: 694 pages in swap cache
Dec 21 09:46:41 mylnx6 kernel: Swap cache stats: add 589182, delete 588488, find 343370/443306
Dec 21 09:46:41 mylnx6 kernel: Free swap = 66723056kB
Dec 21 09:46:41 mylnx6 kernel: Total swap = 67108856kB
Dec 21 09:46:41 mylnx6 kernel: 6291440 pages RAM
Dec 21 09:46:41 mylnx6 kernel: 107316 pages reserved
Dec 21 09:46:41 mylnx6 kernel: 24060 pages shared
Dec 21 09:46:41 mylnx6 kernel: 77648 pages non-shared
Dec 21 09:46:41 mylnx6 kernel: Out of memory: kill process 25011 (oracle) score 8425150 or a child
Dec 21 09:46:41 mylnx6 kernel: Killed process 25011 (oracle)
Dec 21 09:47:20 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
檢查這個系統的記憶體，發現DR環境下，這個伺服器只分配了24G記憶體，而實際生產環境的記憶體為64G（設定了Linux標準大頁，而且SGA_MAX_SIZE大小為32G），而且這個環境是生產環境的“克隆體”，只是由於資源限制的緣故，系統管理員只分配24G記憶體。如下所示：
[[email protected] ~]# free -m
total used free shared buffers cached
Mem: 24156 24033 123 0 0 6
-/+ buffers/cache: 24026 130
Swap: 65535 41 65494
[[email protected] ~]# ps -ef | grep ora_
root 11759 11490 0 16:10 pts/1 00:00:00 grep ora_
[[email protected] ~]# ipcs -m
相關推薦
ORA-00471: DBWR process terminated with error案例
每年年底，系統管理員都要組織一次容災方案的測試、演練。會在一個與生產環境網路隔離的DR環境中，啟動各個“生產環境伺服器”，然後讓各路人員參與其中測試、演練容災方案是否可靠。這次演練中，一臺Oracle資料庫伺服器啟動的時候遇到了問題。如下所示，啟動的時候遇到ORA-0
centos 報錯 “Job for iptables.service failed because the control process exited with error code.”的解決辦法
cau ack res sta ble put use wal ror 原因：因為centos7默認的防火墻是firewalld防火墻，不是使用iptables，因此需要先關閉firewalld服務，或者幹脆使用默認的firewalld防火墻。
操作步驟：
關閉防火墻
1.
centos7啟動iptables時報Job for iptables.service failed because the control process exited with error cod
異常信息 bsp stop input emctl tro stat mct centos7
centos7啟動iptables時報Job for iptables.service failed because the control process exi
Job for network.service failed because the control process exited with error code問題
今天在centOS 7下更改完靜態ip後發現network服務重啟不了，翻遍了網路，嘗試了各種方法，終於解決了。
現把各種解決方法歸納整理，希望能讓後面的同學少走點歪路。。。
首先看問題：執行service network restart命令
centos7下docker啟動失敗--ob for docker.service failed because the control process exited with error code.
今天在配置openshift用docker部署映象的時候，啟動docker發現竟然啟動失敗：
之前是可以啟動的，突然不能啟動了，想到我改了配置檔案：
執行：
vim /etc/sysconfig/docker
可能是家的內容有問題
Centos 7不能上網，Job for iptables.service failed because the control process exited with error code.
一、先檢查自己的網路配置資訊：路徑： vi /etc/sysconfig/network-scripts/ifcfg-eno16777736 如圖
重啟網路配置的命令：service network restart或者是systemctl restart net
伺服器宕機，mysql無法啟動，job for mysql.service failed because the process exited with error code，資料庫備份與恢復
[問題現象]
伺服器在執行過程中，因人為意外導致電源被拔，伺服器宕機，mysql重啟不成功，報錯如下
根據提示，輸入systemctl status mysql.service和journalctl -xe檢視日誌，經過一番百度谷歌折騰也是無果。（很多時候，不能因為突發事件就“病急亂投醫”）
&nb
centos7啟動MySQL時候突然進不去Job for mysqld.service failed because the control process exited with error cod
mysql突然進不去了，不知道什魔鬼
[[email protected] ~]# systemctl start mysqld.service
Job for mysqld.service failed because the control pro
解決mariadb 啟動時報錯Job for mariadb.service failed because the control process exited with error code. Se
錯誤：[[email protected] ~]# systemctl start mariadb.service Job for mariadb.service failed because the control process exited
Problem Solving Process of The terminal process terminated with exit code 1
參考前輩的配置VScode C/C++環境的經驗：
成成賜我力量 bat67
參考之後我的配置
c_cpp_properties.json
"configurations": [
{
"name": "Mac",
Centso7.2安裝mysql報錯Job for mysqld.service failed because the control process exited with error code.
最近安裝了好多環境，忙裡偷閒為安裝的所有環境以及安裝環境的過程中遇到的問題記錄下來。參考來源：https://www.cnblogs.com/qjoanven/p/7699382.html一、centos7.2安裝mysql1、解除安裝系統自帶Mariadb，mariadb是
mysql.service failed because the control process exited with error code問題
騰訊雲ubuntu下配置mysql遠端訪問一開始可以啟動，但後來關掉mysql服務一直重啟失敗
錯誤如下圖所示：
一開始以為是因為程序結束的方式錯誤，所以就重啟了一下伺服器；後來發現重啟伺服器後
Job for iptables.service failed because the control process exit with error code 修復
Job for iptables.service failed because the control process exited with error code. See “systemctl status iptables.service” and “journalctl -xe”
Job for php-fpm.service failed because the control process exited with error code. See "systemctl st
centos7.2環境下安裝php5.6.17。
啟動失敗。
Job for php-fpm.service failed because the control process exited with error code. See "systemctl stat
centos出現這個錯誤：Job for sshd.service failed because the control process exited with error code.
這是因為SElinux啟用了，關閉就好了。在/etc/selinux/config檔案中如下修改，設定SELINUX=disabled：# This file controls the state of SELinux on the system.# SELINUX= can
CentOS7啟動SSH服務報：Job for ssh.service failed because the control process exited with error code.......
CentOS7啟動SSH服務報：Job for ssh.service failed because the control process exited with error codesee systemctl status ssh.service and journalc
salt出錯Job for salt-master.service failed because the control process exited with error code.
問題
&emsap; 在虛擬機器上使用salt的時候，因為一邊看文件一邊學習，突然之前salt就用不了了，然後
ps aux | grep salt
&emsap; 發現salt-maste和salt-minion都在，準備重啟一下sa
解決Job for docker.service failed because the control process exited with error code. See "systemctl s
在ubuntu16.04下安裝安裝docker docker daemon啟不起來，報如下錯誤：
Job for docker.service failed because the control process exited with error code. See "s
【linux錯誤】Job for vsftpd.service failed because the control process exited with error code. See "syst
我的環境：阿里雲centos7.2搭建ftp啟動vsftpd服務，報錯：Job for vsftpd.service failed because the control process exited with error code. See "systemctl statu
init: prefdm main process (1235) terminated with status 1解決辦法
我使用的是centos帶桌面的系統，虛擬機器安裝的，因為這是我的開發環境嘛，所以需要使用些IDE，但是系統總是無緣無故的無法啟動，或者說是崩潰
幸虧虛擬機器系統映象備份了，得以馬上回復，但是也擱不住隔兩天就換呀，這次心一橫，開始解決這個問題，發現問題的過程不多說了，直接薯片解決方