1. 程式人生 > >多麼痛的一次領悟ORA-21561: OID generation failed

多麼痛的一次領悟ORA-21561: OID generation failed

環境為 2個節點的11.2.0.4的RAC環境

重啟異常

資料庫無法啟動,出現報錯如下

在日誌

 /opt/app/grid/11.2.0/log/racoa02/agent/ohasd/oraagent_grid/oraagent_grid.log

2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} checkCrsStat 2 CLSCRS_STAT ret: 184
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} CrsCmd::ClscrsCmdData::destroy
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} CrsCmd::destroy
2018-10-29 23:26:01.841: [ora.evmd][4040279808]{0:0:2} [start] (:CLSN00107:) clsn_agent::start }
2018-10-29 23:26:01.841: [    AGFW][4040279808]{0:0:2} Command: start for resource: ora.evmd 1 1 completed with status: SUCCESS
2018-10-29 23:26:01.841: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.evmd 1 1] ID 4098:438
2018-10-29 23:26:01.841: [ora.evmd][4040279808]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]thread constructor
2018-10-29 23:26:01.842: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]start {
2018-10-29 23:26:01.842: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]start }
2018-10-29 23:26:01.842: [ora.evmd][4040279808]{0:0:2} [check] DaemonAgent::check returned 0
2018-10-29 23:26:01.842: [ora.evmd][4040279808]{0:0:2} [check] Deep check returned 0
2018-10-29 23:26:01.842: [    AGFW][4067276544]{0:0:2} ora.evmd 1 1 state changed from: STARTING to: PARTIAL


2018-10-29 23:26:01.843: [    AGFW][4067276544]{0:0:2} Started implicit monitor for [ora.evmd 1 1] interval=30000 delay=30000
2018-10-29 23:26:01.843: [    AGFW][4067276544]{0:0:2} Agent sending last reply for: RESOURCE_START[ora.evmd 1 1] ID 4098:438
2018-10-29 23:26:01.843: [CLSFRAME][4088489760] TM [MultiThread] is changing desired thread # to 5. Current # is 4
2018-10-29 23:26:03.504: [ora.mdnsd][4040279808]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:05.515: [ora.gpnpd][4069377792]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:06.526: [ora.gipcd][2613040896]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::check prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop  shutdown mode: 4
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop pool pConnxn 00000000
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] clsnInstConnection::makeConnectStr UsrOraEnv  m_oracleHome /opt/app/grid/11.2.0 Crshome /opt/app/grid/11.2.0
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/opt/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/opt/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2018-10-29 23:26:06.751: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop non pool pConnxn 1 d00404c0
2018-10-29 23:26:06.751: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt: server not attached
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] ORA-21561: OID generation failed

2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect1 errcode 21561
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect2 oracleHome /opt/app/grid/11.2.0 oracleSid +ASM1
2018-10-29 23:26:06.763: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt: server not attached
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ORA-21561: OID generation failed

2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect2 errcode 21561
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] clsnUtils::error Exception type=2 string=
ORA-21561: OID generation failed

2018-10-29 23:26:06.772: [    AGFW][4063266560]{0:0:2} sending status msg [ORA-21561: OID generation failed
] for start for resource: ora.asm 1 1
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 1
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection sid  +ASM1, InstConnection d00404c0
2018-10-29 23:26:06.772: [ USRTHRD][4063266560]{0:0:2} InstConnection::breakCall pConnxn:d00404c0  DetachLock:00b01580 m_pSvcH:d00eb5f0
2018-10-29 23:26:06.772: [ USRTHRD][4063266560]{0:0:2} InstConnection:~InstConnection: this d00404c0
2018-10-29 23:26:06.773: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:444
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection delete InstConnection 0x7f1fd00404c0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection freed 1
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop db/asm
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:06.775: [ora.asm][4063266560]{0:0:2} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2018-10-29 23:26:06.775: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop Container->stop, isShutdown = 0
2018-10-29 23:26:06.775: [ USRTHRD][4063266560]{0:0:2} CssData Destructor groupName:HB+ASM
2018-10-29 23:26:06.776: [ USRTHRD][4063266560]{0:0:2} CssData::destroyMutex &m_cssDataLock:d0036a10
2018-10-29 23:26:06.776: [ USRTHRD][4063266560]{0:0:2} CssData::Destructor
2018-10-29 23:26:06.776: [ USRTHRD][4063266560]{0:0:2} Thread:CSS Data Threadthread destructor
2018-10-29 23:26:06.776: [ USRTHRD][4063266560]{0:0:2} Thread:CSS Data Threadthread destructor exit
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: } 
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] clean  }
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] InstAgent::start exception }
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.asm start" encountered the following error: 
ORA-21561: OID generation failed
. For details refer to "(:CLSN00107:)" in "/opt/app/grid/11.2.0/log/racoa01/agent/ohasd/oraagent_grid//oraagent_grid.log".

2018-10-29 23:26:06.776: [    AGFW][4063266560]{0:0:2} sending status msg [CRS-5017: The resource action "ora.asm start" encountered the following error: 
ORA-21561: OID generation failed
. For details refer to "(:CLSN00107:)" in "/opt/app/grid/11.2.0/log/racoa01/agent/ohasd/oraagent_grid//oraagent_grid.log".
] for start for resource: ora.asm 1 1
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [start] (:CLSN00107:) clsn_agent::start }
2018-10-29 23:26:06.776: [    AGFW][4063266560]{0:0:2} Command: start for resource: ora.asm 1 1 completed with status: FAIL
2018-10-29 23:26:06.776: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:444
2018-10-29 23:26:06.776: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::resetConnection  s_statusOfConnectionMap 00b07d28
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::resetConnection sid +ASM1 status  2
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] Gimh::check OH /opt/app/grid/11.2.0 SID +ASM1
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown) exists
2018-10-29 23:26:06.777: [ USRTHRD][4063266560]{0:0:2} Gimh::destructor gimh_dest_query_ctx rc=0
2018-10-29 23:26:06.777: [ USRTHRD][4063266560]{0:0:2} Gimh::destructor gimh_dest_inst_ctx rc=0
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::stopConnection
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] InstAgent::checkState dbType 
2018-10-29 23:26:06.777: [ora.asm][4063266560]{0:0:2} [check] InstAgent::checkState wait 5 seconds
2018-10-29 23:26:06.777: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:444
2018-10-29 23:26:11.777: [ora.asm][4063266560]{0:0:2} [check] InstAgent::check prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:11.777: [    AGFW][4067276544]{0:0:2} ora.asm 1 1 state changed from: STARTING to: PLANNED_OFFLINE
2018-10-29 23:26:11.777: [    AGFW][4067276544]{0:0:2} ora.asm 1 1 state details has changed from:  to: Instance Shutdown
2018-10-29 23:26:11.777: [    AGFW][4067276544]{0:0:2} Agent sending last reply for: RESOURCE_START[ora.asm 1 1] ID 4098:444
2018-10-29 23:26:11.779: [    AGFW][4067276544]{0:0:2} Agent received the message: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:454
2018-10-29 23:26:11.779: [    AGFW][4067276544]{0:0:2} Preparing CLEAN command for: ora.asm 1 1
2018-10-29 23:26:11.779: [    AGFW][4067276544]{0:0:2} ora.asm 1 1 state changed from: PLANNED_OFFLINE to: CLEANING
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] (:CLSN00106:) clsn_agent::clean {
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] clean {
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop_option stop mode immediate option 1
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop { 
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop original reason system do shutdown abort
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::resetConnection  s_statusOfConnectionMap 00b07d28
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::resetConnection sid +ASM1 status  2
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] Gimh::check OH /opt/app/grid/11.2.0 SID +ASM1
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown) exists
2018-10-29 23:26:11.780: [ USRTHRD][4040279808]{0:0:2} Gimh::destructor gimh_dest_query_ctx rc=0
2018-10-29 23:26:11.780: [ USRTHRD][4040279808]{0:0:2} Gimh::destructor gimh_dest_inst_ctx rc=0
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::check 1 prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::checkState dbType 
2018-10-29 23:26:11.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::checkState wait 5 seconds
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::check prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop  shutdown mode: 4
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop pool pConnxn 00000000
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] clsnInstConnection::makeConnectStr UsrOraEnv  m_oracleHome /opt/app/grid/11.2.0 Crshome /opt/app/grid/11.2.0
2018-10-29 23:26:16.780: [ora.asm][4040279808]{0:0:2} [clean] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/opt/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/opt/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2018-10-29 23:26:16.781: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop non pool pConnxn 1 a40717e0
2018-10-29 23:26:16.781: [ora.asm][4040279808]{0:0:2} [clean] InstConnection::connectInt: server not attached
2018-10-29 23:26:16.792: [ora.asm][4040279808]{0:0:2} [clean] ORA-21561: OID generation failed

2018-10-29 23:26:16.793: [ora.asm][4040279808]{0:0:2} [clean] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:16.793: [ora.asm][4040279808]{0:0:2} [clean] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:16.793: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop: connect1 errcode 21561
2018-10-29 23:26:16.793: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop: connect2 oracleHome /opt/app/grid/11.2.0 oracleSid +ASM1
2018-10-29 23:26:16.793: [ora.asm][4040279808]{0:0:2} [clean] InstConnection::connectInt: server not attached
2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] ORA-21561: OID generation failed

2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop: connect2 errcode 21561
2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] clsnUtils::error Exception type=2 string=
ORA-21561: OID generation failed

2018-10-29 23:26:16.803: [    AGFW][4040279808]{0:0:2} sending status msg [ORA-21561: OID generation failed
] for clean for resource: ora.asm 1 1

2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection connection count 1
2018-10-29 23:26:16.803: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection sid  +ASM1, InstConnection a40717e0
2018-10-29 23:26:16.803: [ USRTHRD][4040279808]{0:0:2} InstConnection::breakCall pConnxn:a40717e0  DetachLock:00b01580 m_pSvcH:a40c0520
2018-10-29 23:26:16.803: [ USRTHRD][4040279808]{0:0:2} InstConnection:~InstConnection: this a40717e0
2018-10-29 23:26:16.803: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:454
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection delete InstConnection 0x7f1fa40717e0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection freed 1
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop db/asm
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop Container->stop, isShutdown = 0
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop calling initCssData
2018-10-29 23:26:16.804: [ USRTHRD][4040279808]{0:0:2} Thread:CSS Data Threadthread constructor
2018-10-29 23:26:16.804: [ora.asm][4040279808]{0:0:2} [clean] CssData Constructor groupName:HB+ASM cssRegistrationType:0 memno:-1
2018-10-29 23:26:16.804: [ USRTHRD][4040279808]{0:0:2} CssData::initMutex &m_cssDataLock:a410f070
2018-10-29 23:26:16.805: [ora.asm][4040279808]{0:0:2} [clean] CssGroup::regis registration of groupName:HB+ASM mbrid:-1 m_grpNum:-1 regType:0 succeeded
2018-10-29 23:26:16.805: [ora.asm][4040279808]{0:0:2} [clean] CssData::Constructor regis succeeded groupName:HB+ASM memno:-1
2018-10-29 23:26:16.805: [ora.asm][4040279808]{0:0:2} [clean] CssData::Constructor m_pCssGroup:a403d750 group HB+ASM
2018-10-29 23:26:16.805: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::init m_groupName:HB+ASM m_pCssData a410dfa0
2018-10-29 23:26:16.805: [ USRTHRD][4040279808]{0:0:2} CssData Destructor groupName:HB+ASM
2018-10-29 23:26:16.806: [ USRTHRD][4040279808]{0:0:2} CssData::destroyMutex &m_cssDataLock:a410f070
2018-10-29 23:26:16.806: [ USRTHRD][4040279808]{0:0:2} CssData::Destructor
2018-10-29 23:26:16.806: [ USRTHRD][4040279808]{0:0:2} Thread:CSS Data Threadthread destructor
2018-10-29 23:26:16.806: [ USRTHRD][4040279808]{0:0:2} Thread:CSS Data Threadthread destructor exit
2018-10-29 23:26:16.806: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:16.806: [ora.asm][4040279808]{0:0:2} [clean] InstAgent::stop: } 
2018-10-29 23:26:16.806: [ora.asm][4040279808]{0:0:2} [clean] clean  }
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [clean] (:CLSN00106:) clsn_agent::clean }
2018-10-29 23:26:16.807: [    AGFW][4040279808]{0:0:2} Command: clean for resource: ora.asm 1 1 completed with status: SUCCESS
2018-10-29 23:26:16.807: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:454
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::resetConnection  s_statusOfConnectionMap 00b07d28
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::resetConnection sid +ASM1 status  2
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] Gimh::check OH /opt/app/grid/11.2.0 SID +ASM1
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown) exists
2018-10-29 23:26:16.807: [ USRTHRD][4040279808]{0:0:2} Gimh::destructor gimh_dest_query_ctx rc=0
2018-10-29 23:26:16.807: [ USRTHRD][4040279808]{0:0:2} Gimh::destructor gimh_dest_inst_ctx rc=0
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::stopConnection
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] InstAgent::checkState dbType 
2018-10-29 23:26:16.807: [ora.asm][4040279808]{0:0:2} [check] InstAgent::checkState wait 5 seconds
2018-10-29 23:26:21.807: [ora.asm][4040279808]{0:0:2} [check] InstAgent::check prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:21.807: [    AGFW][4067276544]{0:0:2} ora.asm 1 1 state changed from: CLEANING to: PLANNED_OFFLINE
2018-10-29 23:26:21.808: [    AGFW][4067276544]{0:0:2} Agent sending last reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:454
2018-10-29 23:26:21.814: [    AGFW][4067276544]{0:0:2} Agent received the message: RESOURCE_START[ora.asm 1 1] ID 4098:461
2018-10-29 23:26:21.814: [    AGFW][4067276544]{0:0:2} Preparing START command for: ora.asm 1 1
2018-10-29 23:26:21.814: [    AGFW][4067276544]{0:0:2} ora.asm 1 1 state changed from: PLANNED_OFFLINE to: STARTING
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] (:CLSN00107:) clsn_agent::start {
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] InstAgent::start { 
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::stopConnection
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:21.814: [ USRTHRD][4040279808]{0:0:2} ConnectionPool::~ConnectionPool m_oracleHome:/opt/app/grid/11.2.0, m_oracleSid:+ASM1, m_usrOraEnv:
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::refresh
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::refresh ORACLE_HOME = /opt/app/grid/11.2.0
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] clsnUtils::cmdIdIsStart CmdId:257
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 2 oracle_sid = 
2018-10-29 23:26:21.814: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 4 oracle_sid = 
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] Node 0 doesn't exist
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::getOracleSid: Exception IntException
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12: 
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid oracle_sid = +ASM1
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::refresh ORACLE_SID = +ASM1
2018-10-29 23:26:21.819: [ USRTHRD][4040279808]{0:0:2} Thread:CSS Data Threadthread constructor
2018-10-29 23:26:21.819: [ora.asm][4040279808]{0:0:2} [start] CssData Constructor groupName:HB+ASM cssRegistrationType:0 memno:-1
2018-10-29 23:26:21.819: [ USRTHRD][4040279808]{0:0:2} CssData::initMutex &m_cssDataLock:a410f070
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] CssGroup::regis registration of groupName:HB+ASM mbrid:-1 m_grpNum:-1 regType:0 succeeded
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] CssData::Constructor regis succeeded groupName:HB+ASM memno:-1
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] CssData::Constructor m_pCssGroup:a4068f40 group HB+ASM
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] InstAgent::init m_groupName:HB+ASM m_pCssData a410dfa0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/opt/app/grid/11.2.0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] Utils::getOracleHomeAttrib oracle_home:/opt/app/grid/11.2.0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/opt/app/grid/11.2.0, m_oracleSid:+ASM1, m_usrOraEnv:
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] updating GEN_USR_ORA_INST_NAME to +ASM1
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value +ASM1
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME value +ASM1 retCode 0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] sModifyConfig for +ASM
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/opt/app/grid/11.2.0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] Utils::getOracleHomeAttrib oracle_home:/opt/app/grid/11.2.0
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] sclsnInstAgent::sUpdateOratab oratab is /etc/oratab
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] ConfigFile::setAltName altName = +ASM
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] ConfigFile::setAltValue altValue = /opt/app/grid/11.2.0:N
2018-10-29 23:26:21.820: [ora.asm][4040279808]{0:0:2} [start] sclsnInstAgent::sUpdateOratab CssLock
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] clsnUtils::cmdIdIsStart CmdId:257
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 2 oracle_sid = 
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 4 oracle_sid = 
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] Node 0 doesn't exist
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::getOracleSid: Exception IntException
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12: 
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] AsmAgent:getOracleSid oracle_sid = +ASM1
2018-10-29 23:26:21.821: [ora.asm][4040279808]{0:0:2} [start] ConfigFile::getAltEntry altEntry = +ASM1
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] ConfigFile::getValue name +asm1 value /opt/app/grid/11.2.0:N
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] sclsnInstAgent::sUpdateOratab entry exists with dbName +ASM1 value /opt/app/grid/11.2.0:N
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] sclsnInstAgent::sUpdateOratab CSS unlock
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] clsnInstConnection::makeConnectStr UsrOraEnv  m_oracleHome /opt/app/grid/11.2.0 Crshome /opt/app/grid/11.2.0
2018-10-29 23:26:21.822: [ora.asm][4040279808]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/opt/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/opt/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2018-10-29 23:26:21.823: [ora.asm][4040279808]{0:0:2} [start] Container:start oracle home /opt/app/grid/11.2.0
2018-10-29 23:26:21.823: [ora.asm][4040279808]{0:0:2} [start] InstConnection::connectInt: server not attached
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] ORA-21561: OID generation failed

2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::start: 1 errcode 21561
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::start: untrapped error
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::start: 2 errcode 21561
2018-10-29 23:26:21.834: [ USRTHRD][4040279808]{0:0:2} InstConnection:~InstConnection: this a4076770
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] clean {
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::stop_option stop mode immediate option 1
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::stop { 
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] InstAgent::stop original reason system do shutdown abort
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::resetConnection  s_statusOfConnectionMap 00b07d28
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] ConnectionPool::resetConnection sid +ASM1 status  2
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] Gimh::check OH /opt/app/grid/11.2.0 SID +ASM1
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown) exists
2018-10-29 23:26:21.834: [ora.asm][4040279808]{0:0:2} [start] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2018-10-29 23:26:21.835: [ COMMCRS][2610939648]clsc_connect: (0x7f1fa40e1ad0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

除了報錯,連線資料庫出現異常,異常如下

[[email protected] ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Oct 30 00:47:45 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

connect 

SQL >

這種現象很詭異,在mos上搜索相關文件

Connection Via Oracle Net Fails With Error ORA-21561: OID Generation Failed (文件 ID 559981.1)

Changes

It is likely that a recent upgrade has taken place.
This error is usually the result of a hostname (FQDN or shortname) resolution issue.

Cause

Incorrect /etc/hosts mapping for the client machine or no mapping at all. 

Solution

Check the client's /etc /.../hosts file and verify that the local node is correctly identified with a mapping for both the short and fully qualified name of the host.

對比主機名發現

/etc/hosts 主機名為 racoa01

而 /etc/sysconfig/network 主機名為 racoa01.bjzb

將/etc/sysconfig/network 主機名更改為 racoa01,重啟即可

相關推薦

多麼領悟ORA-21561: OID generation failed

環境為 2個節點的11.2.0.4的RAC環境 重啟異常 資料庫無法啟動,出現報錯如下 在日誌  /opt/app/grid/11.2.0/log/racoa02/agent/ohasd/oraagent_grid/oraagent_grid.log 201

【解決】Mac升級Mojave後用Navicat連線Oracle資料庫報ORA-21561: OID generation failed

一、原因排查 終端執行命令: xxxdeMacBook-Pro:~ xxx$ hostname xxxdeMacBook-Pro.local 發現是 輸出的hostname為 .local 而不是 localhost  檢視 hosts 檔案 : vim /e

WPF資料爬取小工具-某寶推廣位批量生成,及訂單爬取 記:接單最的感悟

專案由來:上月閒來無事接到接到一個單子,自動登入 X寶平臺,然後重定向到指定頁面批量生成推廣位資訊;與此同時自動定時同步訂單資料到需求提供方的Java服務。 當然期間遇到一個小小的問題就是介面樣式的問題,起初使用的winform開發,但是樣式,你懂的,所以後來索性直接使用wpf. 先宣告:這裡只做經驗分享

不能開啟資料庫的處理 ORA-01092: ORACLE instance terminated. Disconnection forced

今天有人說資料庫掛了,請我幫忙看一下。是9i的庫,測試環境的。 啟動報錯如下: [[email protected] oracle]$ sqlplus "/as sysdba" SQL*Plus: Release 9.2.0.4.0 - Production on

sshd啟動報錯,Failed to start OpenSSH server daemon.

sshd -t [[email protected] proj]# sshd -t @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY F

簡單記錄REDO文件損壞報錯 ORA-00333重做日誌讀取塊出錯

clas 後者 利用 實例恢復 poi cancel true cover html 一.故障描寫敘述 首先是實例恢復須要用到的REDO文件損壞 二、解決方法 1.對於非當前REDO或者當前REDO可是無活動事務使用下面CLEAR命令: 用CLEAR命令重建該日誌

內存溢出的分析經歷——thrift帶給我的orz

一個bug 服務端 ide 參數 comment ces 結果 業務 改變 說在前面的話 朋友,你經歷過部署好的服務突然內存溢出嗎? 你經歷過沒有看過Java虛擬機,來解決內存溢出的痛苦嗎? 你經歷過一個BUG,百思不得其解,頭發一根一根脫落的煩惱嗎? 我知道,你有過! 但

ORA-600處理

ORACLE ORA-600 UNDO ORA-00600:internal error code, arguments: [4194], [43], [46], [], [], [], [], []出現這種情況,大多數是因為異常宕機,在啟動的時候報的錯誤。DB 不能啟動。 方法一:使用syste

redo損壞導致ora-600[4000]的恢復

據說今天是光棍節,逢年過節,必有大事。快下班的時候,一位朋友碰到了一個大問題,資料庫伺服器異常斷電重啟以後,資料庫無法啟動,報ora-600[4000]錯誤,嘗試了使用隱藏引數,還是無法開啟。 ORA-00704: bootstrap process failure ORA

ORA-07445[opiaba()+639],ORA-00600[17147]資料庫異常中止故障案例

故障描述: 資料庫版本:11.2.0.3.0(單例項) 作業系統版本:redhat linux 6.5 資料庫在凌晨3點自動關閉,同事發現故障後,立馬手動startup開啟了資料庫,開啟後資料庫執行正常。 故障時資料庫alert日誌如下: Wed Oct 10 02:

多麼領悟:十三起慘痛宕機案例

社群有很多兄弟分享慘痛宕機案例,提醒大家需警惕,以下介紹幾起,滿滿都是血的教訓…… (以下案例來自社群多位會員分享,主要由社群專家孫偉光、崔增順編輯整理) 1.AIX 下 NTP 設定不當導致的多個叢集宕機 事情發生在一段時間之前,接到朋友電話,使用者有三套 oracle rac 叢集執行在 aix

ORA-15064,ORA-12012,ORA-01565

記一次ORACLE報錯的處理 1.1配置 專案 DB Info 型別 RAC Version 12.1.0.2 Sto

ORA-00600,ORA-10173報錯

ORA-00600,ORA-10173 1.1故障環境介紹 專案 DB Info DB型別 RAC Version 12.1.0.2

iOS 控制元件載入圖片不顯示的原因--多麼領悟

在編輯專案程式碼的過程中曾多次出現載入圖片不顯示的情況,有按鈕上的,也有imageView上的,當時稀裡糊塗的換了圖片就可以了,還以為是圖片本身的格式問題,現在終於抓到元凶了。 在Xcode中的Ima

華為8年女碩離職:多麼領悟...

一年前的十月,我離開了華為,不鹹不淡。這個十月,和我一同入職的先生也離開了,心裡的五味雜陳卻異常凶猛的翻湧起來。過去的一年裡,我還是稱華為我們公司,因為我還是華為家屬,生活裡依然瀰漫著熟悉的華為味道,還有機會在上研所美麗的園區溜達。可是從此以後,就真的和這裡沒有關係了,真真的告別了。於是提起筆,整理此刻的

記憶體溢位的分析經歷——thrift帶給我的orz

說在前面的話朋友,你經歷過部署好的服務突然記憶體溢位嗎?你經歷過沒有看過Java虛擬機器,來解決記憶體溢位的痛苦嗎?你經歷過一個BUG,百思不得其解,頭髮一根一根脫落的煩惱嗎?我知道,你有過!但是我還是要來說說我的故事..................背景:有一個專案做一個

impdp匯入資料時的ORA-31696錯誤

今天幫一同事從一個dump檔案中匯入資料到測試庫中,結果死活出現ORA-31696的錯誤: [[email protected]]$ impdp pebank/pebank directory=dumpdir dumpfile=mcj123.1011.dmp

斷電恢復ORA-01033錯誤

客戶的電腦因為頻繁斷電,造成orcle無法連線,報ORA-01033:oracle初始化或者關閉錯誤. 按照傳統方法 進入cmd sqlplus sys/[email protected] as sysdba 可以連線.關閉資料庫 shutdown imm

多麼領悟——計算機組成原理第一講

多麼痛的領悟——計算機組成原理第一講 前言 大家好,我是泰斗賢若如,我又開始更新文章了,本次更新的內容是計算機組成原理,是大學計算機相關專業必須學的,我是大三上學期學的,剛開始學的時候感覺很難,get不到重點,直到學了一遍,被期末考試逼了一遍,我才有所領悟,多麼痛的領悟啊。我打算自己

mapreduce讀取不到輸入文件的問題

mapreduce 過濾器hdfs上輸入文件所在包含兩個目錄,分別是: /20170503/shoplast/ /20170503/shop/但是我想過濾掉shop,只把shoplast作為輸入故我實現了過濾器如下: public static class Fi