pinpoint 修改hbase表TTL值
阿新 • • 發佈：2019-04-29
maps 數據 fault 之前 run grep conf readme compose 參考文檔
https://greatwqs.iteye.com/blog/1741330緣起
Pinpoint接入業務監控後數據量大漲，平均每天Hbase數據增量20G左右，數據量太大，需要對數據進行定期清理，否則監控可用性降低，由於之前環境是由docker-compose部署，查到hbase可以修改表的ttl來清理數據，目前進入pinpoint-hbase容器操作，如果能在hbase表格生成時就修改ttl效果會更佳，該方法需要熟悉docker-compose裏面pinpoint-web及pinpoint-hbase部署方法，後期跟進
操作步驟
查找出數據大的hbase表
[email protected]:/opt/hbase/hbase-1.2.6# ls CHANGES.txt LICENSE.txt README.txt conf hbase-webapps logs LEGAL NOTICE.txt bin docs lib [email protected]:/opt/hbase/hbase-1.2.6# cd bin/ [email protected]:/opt/hbase/hbase-1.2.6/bin# ls draining_servers.rb hbase-jruby rolling-restart.sh get-active-master.rb hbase.cmd shutdown_regionserver.rb graceful_stop.sh hirb.rb start-hbase.cmd hbase local-master-backup.sh start-hbase.sh hbase-cleanup.sh local-regionservers.sh stop-hbase.cmd hbase-common.sh master-backup.sh stop-hbase.sh hbase-config.cmd region_mover.rb test hbase-config.sh region_status.rb thread-pool.rb hbase-daemon.sh regionservers.sh zookeepers.sh hbase-daemons.sh replication [email protected]:/home/pinpoint/hbase/data/default# ls AgentEvent AgentStatV2 ApplicationMapStatisticsCallee_Ver2 ApplicationStatAggre SqlMetaData_Ver2 AgentInfo ApiMetaData ApplicationMapStatisticsCaller_Ver2 ApplicationTraceIndex StringMetaData AgentLifeCycle ApplicationIndex ApplicationMapStatisticsSelf_Ver2 HostApplicationMap_Ver2 TraceV2 [email protected]:/home/pinpoint/hbase/data/default# du -h |grep G 17G ./TraceV2 2.2G ./ApplicationTraceIndex 19G . 24小時產生數據大概20G，發現其中TraceV2及ApplicationTraceIndex數據比較大，設置TTL分別為7Day及14Day
進入hbase修改表ttl
[email protected]:/opt/hbase/hbase-1.2.6/bin# ./hbase shell 2019-04-26 12:31:44,071 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable HBase Shell; enter ‘help<RETURN>‘ for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017 hbase(main):001:0> list TABLE AgentEvent AgentInfo AgentLifeCycle AgentStatV2 ApiMetaData ApplicationIndex ApplicationMapStatisticsCallee_Ver2 ApplicationMapStatisticsCaller_Ver2 ApplicationMapStatisticsSelf_Ver2 ApplicationStatAggre ApplicationTraceIndex HostApplicationMap_Ver2 SqlMetaData_Ver2 StringMetaData TraceV2 15 row(s) in 0.1750 seconds => ["AgentEvent", "AgentInfo", "AgentLifeCycle", "AgentStatV2", "ApiMetaData", "ApplicationIndex", "ApplicationMapStatisticsCallee_Ver2", "ApplicationMapStatisticsCaller_Ver2", "ApplicationMapStatisticsSelf_Ver2", "ApplicationStatAggre", "ApplicationTraceIndex", "HostApplicationMap_Ver2", "SqlMetaData_Ver2", "StringMetaData", "TraceV2"] hbase(main):002:0> describe ‘TraceV2‘ Table TraceV2 is ENABLED TraceV2 COLUMN FAMILIES DESCRIPTION {NAME => ‘S‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, DATA_BLOCK_ENCODING => ‘PREFIX‘, TTL => ‘5184000 SECONDS (60 DAYS)‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 1 row(s) in 0.1000 seconds hbase(main):003:0> disable ‘TraceV2‘ 0 row(s) in 8.3610 seconds hbase(main):004:0> alter ‘TraceV2‘ , {NAME=>‘S‘,TTL=>‘604800‘} Updating all regions with the new schema... 256/256 regions updated. Done. 0 row(s) in 1.9750 seconds hbase(main):001:0> hbase(main):002:0* enable ‘TraceV2‘ 0 row(s) in 28.5440 seconds hbase(main):003:0> describe ‘TraceV2‘ Table TraceV2 is ENABLED TraceV2 COLUMN FAMILIES DESCRIPTION {NAME => ‘S‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, DATA_BLOCK_ENCODING => ‘PREFIX‘, TTL => ‘604800 SECONDS (7 DAYS)‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 1 row(s) in 0.2410 seconds 設置ApplicationTraceIndex的TTL為 14天 hbase(main):004:0> describe ‘ApplicationTraceIndex‘ Table ApplicationTraceIndex is ENABLED ApplicationTraceIndex COLUMN FAMILIES DESCRIPTION {NAME => ‘I‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, DATA_BLOCK_ENCODING => ‘PREFIX‘, TTL => ‘5184000 SECONDS (60 DAYS)‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 1 row(s) in 0.0240 seconds hbase(main):007:0> disable ‘ApplicationTraceIndex‘ 0 row(s) in 2.2970 seconds hbase(main):008:0> alter ‘ApplicationTraceIndex‘ , {NAME=>‘I‘,TTL=>‘1209600‘} Updating all regions with the new schema... 16/16 regions updated. Done. 0 row(s) in 1.9250 seconds hbase(main):009:0> enable ‘ApplicationTraceIndex‘ 0 row(s) in 2.2350 seconds hbase(main):010:0> describe ‘ApplicationTraceIndex‘ Table ApplicationTraceIndex is ENABLED ApplicationTraceIndex COLUMN FAMILIES DESCRIPTION {NAME => ‘I‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, DATA_BLOCK_ENCODING => ‘PREFIX‘, TTL => ‘1209600 SECONDS (14 DAYS)‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 1 row(s) in 0.0290 seconds hbase(main):012:0> major_compact ‘ApplicationTraceIndex‘ 0 row(s) in 0.3740 seconds
備註
major_compact的操作目的 合並文件 清除刪除、過期、多余版本的數據 提高讀寫數據的效率 604800 7day describe ‘TraceV2‘ disable ‘TraceV2‘ alter ‘TraceV2‘ , {NAME=>‘S‘,TTL=>‘604800‘} enable ‘TraceV2‘ disable ‘TraceV2‘ major_compact ‘TraceV2‘ 1209600 14day describe ‘ApplicationTraceIndex‘ disable ‘ApplicationTraceIndex‘ alter ‘ApplicationTraceIndex‘ , {NAME=>‘I‘,TTL=>‘1209600‘} enable ‘ApplicationTraceIndex‘ disable ‘ApplicationTraceIndex‘ major_compact ‘ApplicationTraceIndex‘
pinpoint 修改hbase表TTL值