Elasticsearch 6.5 實現冷熱資料分離
阿新 • • 發佈：2018-12-25
一、安裝Elasticsearch叢集web管理工具Cerebro
1.1 下載
https://github.com/lmenezes/cerebro/releases
[[email protected] software]# wget https://github.com/lmenezes/cerebro/releases/download/v0.8.1/cerebro-0.8.1.zip
1.2 安裝及配置
[[email protected] software]# unzip cerebro-0.8.1.zip [[email protected]software]# ln -s cerebro-0.8.1 cerebro [[email protected] software]# cd cerebro [[email protected] cerebro]# ll total 16 drwxr-xr-x 2 root root 40 Dec 5 10:40 bin drwxr-xr-x 3 root root 103 Dec 5 10:40 conf drwxr-xr-x 2 root root 8192 Dec 5 10:40 lib -rw-r--r-- 1 root root 1081 Jun 20 11:03 README.md [[email protected] cerebro]# cd conf [[email protected] conf]# vim application.conf hosts = [ { host = "http://es-master1.linuxplus.com:9200" name = "linuxplus" auth = { username = "root" password = "123456" } } ] [[email protected] conf]# cd .. [[email protected] cerebro]# nohup ./bin/cerebro -Dhttp.port=1234 -Dhttp.address=IP & [1] 77172
二、配置Elasticsearch實現冷熱資料分離
| 節點名稱 | 伺服器型別 | 儲存資料 |
| es-master1 | SATA | 元資料 |
| es-master2 | ||
| es-master3 | ||
| es-hot1 | SSD | Hot |
| es-hot2 | ||
| es-hot3 | ||
| es-stale1 | SATA | Cold |
| es-stale2 |
2.1 配置Master節點
Master1節點配置（其他節點配置類似）
[[email protected] ~]# cd /etc/elasticsearch/ [[email protected] elasticsearch]# vim elasticsearch.yml cluster.name: linuxplus node.name: es-master1.linuxplus.com node.attr.rack: r6 node.master: true node.data: false path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: ["es-master1.linuxplus.com:9300","es-master2.linuxplus.com:9300","es-master3.linuxplus.com:9300","es-hot1.linuxplus.com:9300","es-hot2.linuxplus.com:9300","es-hot3.linuxplus.com:9300","es-stale1.linuxplus.com:9300","es-stale2.linuxplus.com:9300"] discovery.zen.minimum_master_nodes: 1 bootstrap.system_call_filter: false
2.2 配置Hot節點
Hot1節點配置（其他節點配置類似）
[[email protected] elasticsearch]# vim elasticsearch.yml cluster.name: linuxplus node.name: es-hot1.linuxplus.com #需要修改 node.attr.rack: r1 node.master: false node.data: true path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 10.10.10.24 #需要修改 discovery.zen.ping.unicast.hosts: ["es-master1.linuxplus.com:9300","es-master2.linuxplus.com:9300","es-master3.linuxplus.com:9300"] discovery.zen.minimum_master_nodes: 1 bootstrap.system_call_filter: false node.attr.box_type: hot [[email protected] elasticsearch]# /etc/init.d/elasticsearch start
在Hot節點打tag
node.attr.box_type: hot
2.3 配置Stale節點
Stale1節點配置（其他節點配置類似）
[[email protected] elasticsearch]# vim elasticsearch.yml cluster.name: linuxplus node.name: es-stale1.linuxplus.com node.attr.rack: r1 node.master: false node.data: true path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 10.10.10.27 discovery.zen.ping.unicast.hosts: ["es-master1.linuxplus.com:9300","es-master2.linuxplus.com:9300","es-master3.linuxplus.com:9300"] discovery.zen.minimum_master_nodes: 1 bootstrap.system_call_filter: false node.attr.box_type: cold [[email protected] elasticsearch]# /etc/init.d/elasticsearch start
在Stale節點打tag
node.attr.box_type: cold
三、實現熱資料寫入指定節點
通過Cerebro修改模板新增如下配置
"settings": {
"index": {
"number_of_shards": "5",
"auto_expand_replicas": "0-1",
"routing": {
"allocation": {
"require": {
"box_type": "hot"
}
}
}
}
},
通過shell指令碼將Hot資料（保留7天）遷移到Stale
#!/bin/bash
Time=$(date -d "1 week ago" +"%Y.%m.%d")
Hostname=$(hostname)
arr=("cron" "messages" "secure" "tomcat" "nginx-access" "nginxtcp" "nginxerror" "windows" ".monitoring-es-6" ".monitoring-beats-6" ".monitoring-kibana-6" ".monitoring-logstash-6" "metricbeat-6.5.3")
for var in ${arr[@]}
do
curl -H "Content-Type: application/json" -XPUT http://$Hostname:9200/$var-$Time/_settings?pretty -d'
{
"settings": {
"index.routing.allocation.require.box_type": "cold"
}
}'
done