hbase 12.1 regionserver 宕機報錯記錄——1
阿新 • • 發佈：2019-01-25
"Append_min" : 0,
"Append_max" : 0,
"Append_mean" : 0.0,
"Append_median" : 0.0,
"Append_75th_percentile" : 0.0,
"Append_90th_percentile" : 0.0,
"Append_95th_percentile" : 0.0,
"Append_99th_percentile" : 0.0,
"SplitTime_num_ops" : 14,
"SplitTime_min" : 1198,
"SplitTime_max" : 4528,
"SplitTime_mean" : 2753.4285714285716,
"SplitTime_median" : 2668.0,
"SplitTime_75th_percentile" : 3402.0,
"SplitTime_90th_percentile" : 3531.0,
"SplitTime_95th_percentile" : 3531.0,
"SplitTime_99th_percentile" : 3531.0,
"SplitTime_TimeRangeCount_1000-3000" : 9,
"SplitTime_TimeRangeCount_3000-10000" : 5
} ]
}
2016-07-09 16:11:18,157 INFO [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase regionserver:16020-0x154fb34c4132eb5 received expired from ZooKeeper, aborting
2016-07-09 16:11:18,158 INFO [regionserver/ut09/10.80.248.19:16020.logRoller] regionserver.LogRoller: LogRoller exiting.
2016-07-09 16:11:18,158 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-07-09 16:11:18,162 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2016-07-09 16:11:18,163 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Stopping infoServer
2016-07-09 16:11:18,413 INFO [SplitLogWorker-ut09:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
2016-07-09 16:11:18,419 INFO [SplitLogWorker-ut09:16020] regionserver.SplitLogWorker: SplitLogWorker ut09,16020,1467006160072 exiting
2016-07-09 16:11:18,570 INFO [regionserver/ut09/10.80.248.19:16020] mortbay.log: Stopped
2016-07-09 16:11:18,676 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2016-07-09 16:11:18,676 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2016-07-09 16:11:18,676 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2016-07-09 16:11:18,722 INFO [regionserver/ut09/10.80.248.19:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2016-07-09 16:11:18,772 INFO [regionserver/ut09/10.80.248.19:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2016-07-09 16:11:18,795 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: aborting server ut09,16020,1467006160072
2016-07-09 16:11:19,381 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Waiting on 15 regions to close
2016-07-09 16:11:19,406 INFO [StoreCloserThread-wlan_log,51:93:32 40-70-6102|HWES242999910179800,1467911977102.9a12d3f1055f50a02f54bf6a89edbd52.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,408 INFO [StoreCloserThread-wlan_log,62:55:21 60-70-6102|HWSZ2329999107291552,1468031436966.519c62fce7dad85485309852aeb8278e.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,408 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 127460728
2016-07-09 16:11:19,408 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 103098896
2016-07-09 16:11:19,408 INFO [StoreCloserThread-wlan_log,63:50:32 40-70-6102|HWSY2279999104891720,1468035975548.ca2997727d29175e748e4681c4f4d4a7.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,62:55:21 60-70-6102|HWSZ2329999107291552,1468031436966.519c62fce7dad85485309852aeb8278e.
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,51:93:32 40-70-6102|HWES242999910179800,1467911977102.9a12d3f1055f50a02f54bf6a89edbd52.
2016-07-09 16:11:19,409 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 52396816
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,63:50:32 40-70-6102|HWSY2279999104891720,1468035975548.ca2997727d29175e748e4681c4f4d4a7.
2016-07-09 16:11:19,410 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,411 INFO [StoreCloserThread-wlan_log,65:33:81 40-70-6102|HWHS24799991043992,1468038959115.3c3e34a9b09d4f37dd592a7210c9ac0f.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,411 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,424 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 21415000
2016-07-09 16:11:19,424 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,425 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,65:33:81 40-70-6102|HWHS24799991043992,1468038959115.3c3e34a9b09d4f37dd592a7210c9ac0f.
2016-07-09 16:11:19,477 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.
2016-07-09 16:11:19,479 INFO [StoreCloserThread-wlan_log,61:63:10 60-70-6102|HWJZ2339999104592144,1468031436966.6e445f6fa6fd26b709d7c8861bedd84b.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,481 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 41586064
2016-07-09 16:11:19,482 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,61:63:10 60-70-6102|HWJZ2339999104592144,1468031436966.6e445f6fa6fd26b709d7c8861bedd84b.
2016-07-09 16:11:19,483 INFO [StoreCloserThread-wlan_log,64:41:70 50-70-6102|HWXN2239999109892408,1468035975548.c8fff7795e3d5f9d2fe56c25f8d4a2ca.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,485 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 37542896
2016-07-09 16:11:19,485 INFO [StoreCloserThread-wlan_log,55:73:91 40-70-6102|HWXY241999911229416,1467908535398.ccdd6f99f4850977e462e0549ebd3411.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,486 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,64:41:70 50-70-6102|HWXN2239999109892408,1468035975548.c8fff7795e3d5f9d2fe56c25f8d4a2ca.
2016-07-09 16:11:19,486 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 50564544
2016-07-09 16:11:19,487 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,55:73:91 40-70-6102|HWXY241999911229416,1467908535398.ccdd6f99f4850977e462e0549ebd3411.
2016-07-09 16:11:19,492 INFO [StoreCloserThread-wlan_log,53:82:41 40-70-6102|HWXG2619999116991232,1467908535398.5a3fa0f519ee2edc09f1f716453dd022.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,493 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 42498488
2016-07-09 16:11:19,494 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,53:82:41 40-70-6102|HWXG2619999116991232,1467908535398.5a3fa0f519ee2edc09f1f716453dd022.
2016-07-09 16:11:19,494 INFO [StoreCloserThread-wlan_log,70:25:91 60-70-6102|HWXN2329999105792054,1468038959115.74cddc0039296df89941f83157147e53.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,495 INFO [StoreCloserThread-wlan_log,50:10:10 60-70-6102|HWQJ223999910239177,1467911977102.00ee2a8c9da4c3e4fcb8c3e18c0b6ca5.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,495 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 1459816
2016-07-09 16:11:19,496 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 131093056
2016-07-09 16:11:19,496 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,70:25:91 60-70-6102|HWXN2329999105792054,1468038959115.74cddc0039296df89941f83157147e53.
2016-07-09 16:11:19,497 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,50:10:10 60-70-6102|HWQJ223999910239177,1467911977102.00ee2a8c9da4c3e4fcb8c3e18c0b6ca5.
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,498 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,503 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,501 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,503 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.
2016-07-09 16:11:19,503 INFO [StoreCloserThread-wlan_log,71:21:02 40-70-6102|HWES24399991017984,1467678944057.c321c0d12cb9817c6676b92c8954ba29.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,503 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 71103616
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,71:21:02 40-70-6102|HWES24399991017984,1467678944057.c321c0d12cb9817c6676b92c8954ba29.
2016-07-09 16:11:19,581 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: stopping server ut09,16020,1467006160072; all regions closed.
2016-07-09 16:11:19,637 WARN [regionserver/ut09/10.80.248.19:16020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /hbase/WALs/ut09,16020,1467006160072-splitting/ut09%2C16020%2C1467006160072.default.1468051308829 (inode 849427): File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1100162849_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3443)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3345)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:758)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:515)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy16.getAdditionalDatanode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:443)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy17.getAdditionalDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy18.getAdditionalDatanode(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:984)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1160)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
2016-07-09 16:11:19,699 ERROR [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Shutdown / close of WAL failed: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/WALs/ut09,16020,1467006160072-splitting/ut09%2C16020%2C1467006160072.default.1468051308829 (inode 849427): File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1100162849_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3443)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3345)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:758)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:515)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2016-07-09 16:11:19,709 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.Leases: regionserver/ut09/10.80.248.19:16020 closing leases
2016-07-09 16:11:19,710 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.Leases: regionserver/ut09/10.80.248.19:16020 closed leases
2016-07-09 16:11:20,044 INFO [regionserver/ut09/10.80.248.19:16020] hbase.ChoreService: Chore service for: ut09,16020,1467006160072 had [[ScheduledChore: Name: MovedRegionsCleaner for region ut09,16020,1467006160072 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: ut09,16020,1467006160072-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS]] on shutdown
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2016-07-09 16:11:21,387 INFO [regionserver/ut09/10.80.248.19:16020.leaseChecker] regionserver.Leases: regionserver/ut09/10.80.248.19:16020.leaseChecker closing leases
2016-07-09 16:11:21,389 INFO [regionserver/ut09/10.80.248.19:16020.leaseChecker] regionserver.Leases: regionserver/ut09/10.80.248.19:16020.leaseChecker closed leases
2016-07-09 16:11:24,409 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system...
2016-07-09 16:11:24,409 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped.
2016-07-09 16:11:24,909 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
2016-07-09 16:11:24,910 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-07-09 16:11:24,910 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started
2016-07-09 16:11:35,085 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2016-07-09 16:11:35,228 WARN [regionserver/ut09/10.80.248.19:16020] zookeeper.ZKUtil: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/ut09,16020,1467006160072
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:309)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2153)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1084)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:35,228 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.ZooKeeperWatcher: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:309)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2153)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1084)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:35,234 INFO [regionserver/ut09/10.80.248.19:16020] ipc.RpcServer: Stopping server on 16020
2016-07-09 16:11:35,236 INFO [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2016-07-09 16:11:35,277 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-07-09 16:11:35,277 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2016-07-09 16:11:51,590 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2016-07-09 16:11:51,590 WARN [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1418)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:51,590 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: stopping server ut09,16020,1467006160072; zookeeper connection closed.
2016-07-09 16:11:51,591 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: regionserver/ut09/10.80.248.19:16020 exiting
2016-07-09 16:11:53,809 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2667)
2016-07-09 16:11:54,380 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShu
2016-07-09 16:11:54,380 INFO [Thread-5] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2016-07-09 16:11:54,408 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook finished.
"Append_max" : 0,
"Append_mean" : 0.0,
"Append_median" : 0.0,
"Append_75th_percentile" : 0.0,
"Append_90th_percentile" : 0.0,
"Append_95th_percentile" : 0.0,
"Append_99th_percentile" : 0.0,
"SplitTime_num_ops" : 14,
"SplitTime_min" : 1198,
"SplitTime_max" : 4528,
"SplitTime_mean" : 2753.4285714285716,
"SplitTime_median" : 2668.0,
"SplitTime_75th_percentile" : 3402.0,
"SplitTime_90th_percentile" : 3531.0,
"SplitTime_95th_percentile" : 3531.0,
"SplitTime_99th_percentile" : 3531.0,
"SplitTime_TimeRangeCount_1000-3000" : 9,
"SplitTime_TimeRangeCount_3000-10000" : 5
} ]
}
2016-07-09 16:11:18,157 INFO [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase regionserver:16020-0x154fb34c4132eb5 received expired from ZooKeeper, aborting
2016-07-09 16:11:18,158 INFO [regionserver/ut09/10.80.248.19:16020.logRoller] regionserver.LogRoller: LogRoller exiting.
2016-07-09 16:11:18,158 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-07-09 16:11:18,162 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2016-07-09 16:11:18,163 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Stopping infoServer
2016-07-09 16:11:18,413 INFO [SplitLogWorker-ut09:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
2016-07-09 16:11:18,419 INFO [SplitLogWorker-ut09:16020] regionserver.SplitLogWorker: SplitLogWorker ut09,16020,1467006160072 exiting
2016-07-09 16:11:18,570 INFO [regionserver/ut09/10.80.248.19:16020] mortbay.log: Stopped
2016-07-09 16:11:18,676 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2016-07-09 16:11:18,676 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2016-07-09 16:11:18,676 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2016-07-09 16:11:18,722 INFO [regionserver/ut09/10.80.248.19:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2016-07-09 16:11:18,772 INFO [regionserver/ut09/10.80.248.19:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2016-07-09 16:11:18,795 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: aborting server ut09,16020,1467006160072
2016-07-09 16:11:19,381 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Waiting on 15 regions to close
2016-07-09 16:11:19,406 INFO [StoreCloserThread-wlan_log,51:93:32 40-70-6102|HWES242999910179800,1467911977102.9a12d3f1055f50a02f54bf6a89edbd52.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,408 INFO [StoreCloserThread-wlan_log,62:55:21 60-70-6102|HWSZ2329999107291552,1468031436966.519c62fce7dad85485309852aeb8278e.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,408 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 127460728
2016-07-09 16:11:19,408 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 103098896
2016-07-09 16:11:19,408 INFO [StoreCloserThread-wlan_log,63:50:32 40-70-6102|HWSY2279999104891720,1468035975548.ca2997727d29175e748e4681c4f4d4a7.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,62:55:21 60-70-6102|HWSZ2329999107291552,1468031436966.519c62fce7dad85485309852aeb8278e.
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,51:93:32 40-70-6102|HWES242999910179800,1467911977102.9a12d3f1055f50a02f54bf6a89edbd52.
2016-07-09 16:11:19,409 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 52396816
2016-07-09 16:11:19,409 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,63:50:32 40-70-6102|HWSY2279999104891720,1468035975548.ca2997727d29175e748e4681c4f4d4a7.
2016-07-09 16:11:19,410 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,411 INFO [StoreCloserThread-wlan_log,65:33:81 40-70-6102|HWHS24799991043992,1468038959115.3c3e34a9b09d4f37dd592a7210c9ac0f.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,411 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,424 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 21415000
2016-07-09 16:11:19,424 INFO [StoreCloserThread-viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,425 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,65:33:81 40-70-6102|HWHS24799991043992,1468038959115.3c3e34a9b09d4f37dd592a7210c9ac0f.
2016-07-09 16:11:19,477 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed viewlog2,25434148660|2016-01-06 14:16:44,1466758620104.3c6031bb034f773ec97762af5c83fb26.
2016-07-09 16:11:19,479 INFO [StoreCloserThread-wlan_log,61:63:10 60-70-6102|HWJZ2339999104592144,1468031436966.6e445f6fa6fd26b709d7c8861bedd84b.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,481 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 41586064
2016-07-09 16:11:19,482 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,61:63:10 60-70-6102|HWJZ2339999104592144,1468031436966.6e445f6fa6fd26b709d7c8861bedd84b.
2016-07-09 16:11:19,483 INFO [StoreCloserThread-wlan_log,64:41:70 50-70-6102|HWXN2239999109892408,1468035975548.c8fff7795e3d5f9d2fe56c25f8d4a2ca.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,485 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 37542896
2016-07-09 16:11:19,485 INFO [StoreCloserThread-wlan_log,55:73:91 40-70-6102|HWXY241999911229416,1467908535398.ccdd6f99f4850977e462e0549ebd3411.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,486 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,64:41:70 50-70-6102|HWXN2239999109892408,1468035975548.c8fff7795e3d5f9d2fe56c25f8d4a2ca.
2016-07-09 16:11:19,486 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 50564544
2016-07-09 16:11:19,487 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,55:73:91 40-70-6102|HWXY241999911229416,1467908535398.ccdd6f99f4850977e462e0549ebd3411.
2016-07-09 16:11:19,492 INFO [StoreCloserThread-wlan_log,53:82:41 40-70-6102|HWXG2619999116991232,1467908535398.5a3fa0f519ee2edc09f1f716453dd022.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,493 ERROR [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Memstore size is 42498488
2016-07-09 16:11:19,494 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed wlan_log,53:82:41 40-70-6102|HWXG2619999116991232,1467908535398.5a3fa0f519ee2edc09f1f716453dd022.
2016-07-09 16:11:19,494 INFO [StoreCloserThread-wlan_log,70:25:91 60-70-6102|HWXN2329999105792054,1468038959115.74cddc0039296df89941f83157147e53.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,495 INFO [StoreCloserThread-wlan_log,50:10:10 60-70-6102|HWQJ223999910239177,1467911977102.00ee2a8c9da4c3e4fcb8c3e18c0b6ca5.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,495 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 1459816
2016-07-09 16:11:19,496 ERROR [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Memstore size is 131093056
2016-07-09 16:11:19,496 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,70:25:91 60-70-6102|HWXN2329999105792054,1468038959115.74cddc0039296df89941f83157147e53.
2016-07-09 16:11:19,497 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed wlan_log,50:10:10 60-70-6102|HWQJ223999910239177,1467911977102.00ee2a8c9da4c3e4fcb8c3e18c0b6ca5.
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,497 INFO [StoreCloserThread-viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,498 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed viewlog2,22831642660|2016-01-01 00:22:12,1466758620104.d20897c8307b64208cfb8cf43c68d445.
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf1
2016-07-09 16:11:19,500 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,503 INFO [StoreCloserThread-viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,501 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf2
2016-07-09 16:11:19,503 INFO [StoreCloserThread-viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.-1] regionserver.HStore: Closed cf3
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-1] regionserver.HRegion: Closed viewlog2,715665240670|2016-01-05 22:50:22,1466758078110.e965df4398ced3b2cdd5c7d9bc23875e.
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-2] regionserver.HRegion: Closed viewlog2,31459301570|2016-01-02 21:49:47,1466758360450.00764b3a75f2c4b29b997d803f473e2e.
2016-07-09 16:11:19,503 INFO [StoreCloserThread-wlan_log,71:21:02 40-70-6102|HWES24399991017984,1467678944057.c321c0d12cb9817c6676b92c8954ba29.-1] regionserver.HStore: Closed cf
2016-07-09 16:11:19,503 ERROR [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Memstore size is 71103616
2016-07-09 16:11:19,503 INFO [RS_CLOSE_REGION-ut09:16020-0] regionserver.HRegion: Closed wlan_log,71:21:02 40-70-6102|HWES24399991017984,1467678944057.c321c0d12cb9817c6676b92c8954ba29.
2016-07-09 16:11:19,581 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: stopping server ut09,16020,1467006160072; all regions closed.
2016-07-09 16:11:19,637 WARN [regionserver/ut09/10.80.248.19:16020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /hbase/WALs/ut09,16020,1467006160072-splitting/ut09%2C16020%2C1467006160072.default.1468051308829 (inode 849427): File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1100162849_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3443)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3345)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:758)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:515)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy16.getAdditionalDatanode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:443)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy17.getAdditionalDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy18.getAdditionalDatanode(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:984)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1160)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
2016-07-09 16:11:19,699 ERROR [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Shutdown / close of WAL failed: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/WALs/ut09,16020,1467006160072-splitting/ut09%2C16020%2C1467006160072.default.1468051308829 (inode 849427): File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1100162849_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3443)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3345)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:758)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:515)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2016-07-09 16:11:19,709 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.Leases: regionserver/ut09/10.80.248.19:16020 closing leases
2016-07-09 16:11:19,710 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.Leases: regionserver/ut09/10.80.248.19:16020 closed leases
2016-07-09 16:11:20,044 INFO [regionserver/ut09/10.80.248.19:16020] hbase.ChoreService: Chore service for: ut09,16020,1467006160072 had [[ScheduledChore: Name: MovedRegionsCleaner for region ut09,16020,1467006160072 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: ut09,16020,1467006160072-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS]] on shutdown
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2016-07-09 16:11:20,045 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2016-07-09 16:11:21,387 INFO [regionserver/ut09/10.80.248.19:16020.leaseChecker] regionserver.Leases: regionserver/ut09/10.80.248.19:16020.leaseChecker closing leases
2016-07-09 16:11:21,389 INFO [regionserver/ut09/10.80.248.19:16020.leaseChecker] regionserver.Leases: regionserver/ut09/10.80.248.19:16020.leaseChecker closed leases
2016-07-09 16:11:24,409 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system...
2016-07-09 16:11:24,409 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped.
2016-07-09 16:11:24,909 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
2016-07-09 16:11:24,910 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-07-09 16:11:24,910 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started
2016-07-09 16:11:35,085 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2016-07-09 16:11:35,228 WARN [regionserver/ut09/10.80.248.19:16020] zookeeper.ZKUtil: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/ut09,16020,1467006160072
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:309)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2153)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1084)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:35,228 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.ZooKeeperWatcher: regionserver:16020-0x154fb34c4132eb5, quorum=10.80.248.12:2181,ut01:2181,ut03:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:309)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2153)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1084)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:35,234 INFO [regionserver/ut09/10.80.248.19:16020] ipc.RpcServer: Stopping server on 16020
2016-07-09 16:11:35,236 INFO [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2016-07-09 16:11:35,277 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-07-09 16:11:35,277 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2016-07-09 16:11:51,590 ERROR [regionserver/ut09/10.80.248.19:16020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2016-07-09 16:11:51,590 WARN [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ut09,16020,1467006160072
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1418)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092)
at java.lang.Thread.run(Thread.java:745)
2016-07-09 16:11:51,590 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: stopping server ut09,16020,1467006160072; zookeeper connection closed.
2016-07-09 16:11:51,591 INFO [regionserver/ut09/10.80.248.19:16020] regionserver.HRegionServer: regionserver/ut09/10.80.248.19:16020 exiting
2016-07-09 16:11:53,809 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2667)
2016-07-09 16:11:54,380 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShu
2016-07-09 16:11:54,380 INFO [Thread-5] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2016-07-09 16:11:54,408 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook finished.
相關推薦
hbase 12.1 regionserver 宕機報錯記錄——1
"Append_min" : 0, "Append_max" : 0, "Append_mean" : 0.0, "Append_median" : 0.0, "Append_75th_percentile" : 0.0, "
Spring 報錯記錄1
2014-09-19 15:38:17,615 DEBUG [org.springframework.web.servlet.handler.BeanNameUrlHandlerMapping] - Rejected bean name 'org.springframewo
hbase regionserver 無法啟動報錯
某天發現，某個節點的regionserver以外關閉，並且在啟動以後立即會關閉 檢視hbase的region server日誌， Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apa
遷移hbase的快照到新集群后RegionServer無法啟動,報錯failed open of region
錯誤日誌摘要：2018-03-12 17:05:29,608 ERROR [RS_OPEN_REGION-our_ambari_clustergn-a05044c6-core-1-003:16020-15] handler.OpenRegionHandler: Failed
saltstack api wheel模塊報錯HTTP/1.1 401 Unauthorized
saltstack api saltapi salt-api報錯 當使用saltstack api調用wheel模塊的時候會出現沒有權限的報錯[[email protected]/* */ ~]# curl -k -v https://localhost:8000 -H "Ac
maven編譯報錯 -source 1.7 中不支持 lambda 表達式
表達 alt 服務 config img comm plugin ava ror Maven項目編譯失敗： [ERROR] COMPILATION ERROR : [INFO] --------------------------------------------
facebook graph api 報錯SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)')
asa res http ner none object odin bundle dap 使用facebook graph api，報錯如下 一開始以為是https證書驗證失敗，查了一下午源碼，沒有看到問題，於是把Python27\lib\site-packages\re
克隆kvm虛擬機報錯ImportError: No module named 'requests.packages.urllib3'
http VM克隆 color quest TE kvm虛擬 vmdk ase exc 2018-06-21 更新系統造成kvm克隆命令報錯 virt-clone -o centos-7-update-clone -n centos-7-maven-test -f
Kettle7.1在window啟動報錯
cep 實驗 txt文件 tegra info level toc 2.0 control 實驗環境： window10 x64 kettle7.1 pdi-ce-7.1.0.0-12.zip 錯誤現象： a java exception has occurred
maven 報錯 -source 1.5 中不支持 diamond 運算符
art apache 執行 解決 支持 onf plugins 編譯 1.7 maven 執行install 報-source 1.5 中不支持 diamond 運算符 解決辦法， 指定編譯版本 <build> <plugins>
東軟HIS系統_打印發票提示“打印機報錯！對訪問XXX的設置無效”解決辦法
com 5.4 報錯 jpg 添加 圖片 技術 bubuko 分享圖片 發票打印報錯 添加打印機，端口跟物理打印機同一個。 添加 MZJSFP,ZYJSFP,YJJFP三個打印機 打印發票錯位 設置自定義紙張 MZJSFP 寬30.40CM 高12.94CM ZYJSFP
tp5報錯筆記(1)
碰到錯誤的時候沒有記錄,都是後面看日誌的時候回想的,如果有描述錯誤的請指出來,我改正.謝謝! error: Provisional headers are shown 錯誤原因:ajax上傳表單檔案是沒有關掉預設的表單請求, 解決方法:ajax請求後加return fa
tf.reshape()的Tensorflow 1.0 版本問題報錯
tf.reverse tf.reverse之前會用一維 bool 張量控制要顛倒哪些維度。現在我們使用軸索引張量。 例如，tf.reverse(a, [True, False, True]) 現在必須是 tf.reve
mysql報錯Expression #1 of SELECT list is not in GROUP BY clause and contains non
專案執行得好好的，換了個數據庫伺服器後就報這個錯了 版本原因 記錄解決辦法 //執行語句 set global sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZER
spring boot 報錯（1）
The content of element type "configuration" must match "(properties?,settings?,typeAliases?,typeHandlers?,objectFactory?... 報錯 如圖： 百度搜
Android問題：報錯Index -1 requested, with a size of 1
使用Cursor使，讀取裡面的資料用到getColumnIndex()時報錯： Index -1 requested, with a size of 1 仔細閱讀過C
webserver報錯Index: 1, Size: 1
/** * 登入獲取 * @return */ public String login() { String loginName = properties.getString("loginName"); String loginPass =
使用最新版本MySQL8.0.12報錯記錄
使用最新版本MySQL報錯，MySQL版本8.0.12。 報錯1： Sun Oct 14 00:45:30 CST 2018 WARN: Establishing SSL connection without server's identity verificatio
使用gSoap根據wsdl檔案生成標頭檔案報錯 Error -1 fault: SOAP-ENV:Client [no subcode]
近來我在linux上使用gSoap，根據客戶提供的WSDL檔案生成標頭檔案的時候，不知什麼原因老報錯，資訊如下： [[email protected] usertools]# wsdl2h -o AAAserver.h AAAserver.wsdl ** T
tensorflow1.12 多GPU協同訓練報錯tensorflow.python.framework.errors_impl.NotFoundError: libnccl.so.2
tensroflow為了提高多模型訓練速度，需要多個GPU同時工作，而且我們一般使用的工作站都是8塊tesla K80，如果能將8塊顯示卡的計算力充分利用起來，將會大大提高模型訓練的速度，縮短模型訓練時間。 這幾天看到tensorflow的mor