> Elasticsearch 是一個可用於分散式以及符合RESTful 風格的搜尋和資料分析引擎。—— [Elastic Stack 官網](https://www.elastic.co/cn/elasticsearch/)  ### 搭建Elasticsearch的“那些事兒"  有一天，在黃金梅麗號的甲板上，韋柏告訴薩博，需要在接下來的專案開發過程中，運用到Elasticsearch資料庫，主要用於黃金梅麗號上的各種裝置採集資料實時查詢，或許後期還會運用於分散式日誌系統的搭建運用等，讓薩博先做一次技術預研。於是，在薩博查詢大量的資料發現，關於Elasticsearch資料庫的搭建，網上的資料幾乎是千篇一律，或者多數都是沒有一個完整的流程。甚至，還發現，對於Elasticsearch資料庫實際應用方面，大多數海賊只侷限於ELK等這樣的情況，就像是大多數海賊提到Redis，就只覺得這個玩意兒，只能用於快取層面，但是實際上人家的功能強大到超出了海賊的想象空間。甚至於，薩博在一個阿里巴巴的地方，找到了關於Elasticsearch資料庫免費試用的資源，但是對於免費的午餐，也許那一天收費之後，那就只能用一首“浪浪”來祭奠這萬惡的黃金之惡。於是在薩博精心研究和分析發現，除了傳統部署Elasticsearch資料庫之外，有個叫Docker 的東西，除了能夠穩定執行Elasticsearch資料庫之外，還能節省不少的伺服器記憶體。於是呀，薩博就開始了探索在Docker上搭建Elasticsearch之之路...... ### Docker部署Elasticsearch的“那些事兒" #### 1.基礎先決條件 阿里雲或者虛擬機器VM主機一個，並且自行裝有Docker容器：  >ps[⚠️注意事項]： >對於Docker安裝請參考：[Linux環境基於CentOS7 搭建部署Docker容器](https://blog.csdn.net/u012103898/article/details/102567115) #### 2.最大檔案開啟數 [2.1]引數優化：ulimit -a |grep open ```yml [root@centos-meteor ~]# ulimit -a |grep open open files (-n) 65535 [root@centos-meteor ~]# ``` [2.2] 修改配置檔案： /etc/security/limits.conf ```yml #在最下邊新增如下兩行 * soft nofile 65536 * hard nofile 65536 * soft nproc 2048 * hard nproc 4096 ``` [2.3] 確認是否修改： ulimit -a |grep open ```yml [root@centos-meteor ~]# ulimit -a |grep open open files (-n) 65535 [root@centos-meteor ~]# ``` #### 3.調整系統程序數 修改方法：vim /etc/security/limits.d/20-nproc.conf ```yml #調整成以下配置 * soft nproc 4096 root soft nproc unlimited ``` #### 4.調整虛擬記憶體和最大併發連線數 虛擬記憶體（Virtual memory）->vm.max_map_count Elasticsearch預設使用 hybrid mmapfs / niofs 目錄來儲存索引。預設作業系統對mmap計數的限制太低，可能引發記憶體不足的異常 [1]臨時解決方法： ```yml sysctl –w vm.max_map_count=262144 sysctl –w vm.max_map_count=262144 ``` 如果不優化會出現啟動Elasticsearch的時候會出現如下提示： ```yml max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144] ``` [2]永久生效方案：更新系統檔案/etc/sysctl.conf的vm.max_map_count欄位 執行命令：vim /etc/sysctl.conf ```yml [root@turbo-master ~]# vim /etc/sysctl.conf [root@turbo-master ~]# #Settings Elasticsearch Virtual Memory—>655360 vm.max_map_count=655360 fs.file-max=655360 vm.overcommit_memory=1 ``` 載入到系統：sysctl -p --load /etc/sysctl.conf ```yml [root@centos-meteor elasticsearch]# sysctl -p --load /etc/sysctl.conf vm.swappiness = 0 net.ipv4.neigh.default.gc_stale_time = 120 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_announce = 2 net.ipv4.tcp_max_tw_buckets = 5000 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_synack_retries = 2 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 kernel.sysrq = 1 vm.max_map_count = 655360 fs.file-max = 655360 vm.overcommit_memory = 1 [root@centos-meteor elasticsearch]# ``` #### 5.關閉elasticsearch自檢測 在elasticsearch.yml中新增配置項： ```yml bootstrap.memory_lock: false bootstrap.system_call_filter: false ``` >ps[⚠️注意事項]： >主要是避免以下問題： >[1]最大檔案開啟數: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536] >[2]調整虛擬記憶體和最大併發連線數: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] >[3]關閉elasticsearch自檢測: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk >[4]調整系統程序數: max number of threads [1024] for user [lish] likely too low, increase to at least [2048] ### 實戰部署ElasticSearch服務 >Docker 部署應用服務的基本步驟：Search[查詢映象]->Pull[拉取映象]->Run[部署映象] 1.查詢Elasticsearch 映象： ``` docker search elasticsearch ```  >ps[注意事項]： >1. 一般拉取映象資源都是從Docker官方倉庫[[docker-hub](https://hub.docker.com/)]拉取，或者自己構建的Docker雲倉庫[aliyun-docker](https://cr.console.aliyun.com/cn-shanghai/instances/repositories) 等 >2. 本教程選取的ELK映象均是基於ELK官方Docker倉庫[elastic-io](https://www.docker.elastic.co/) 2.拉取Elasticsearch 映象： ``` docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1 ```  >ps[注意事項]： >1.本教程採用7.3.x版本，目前最新版本7.4.x[主要用7.3.x版本在阿里雲搭建過，避免入坑問題] >2.拉取的過程中可能會出現[net/http: TLS handshake timeout]問題,多嘗試幾次,主要是網路頻寬限制問題 3.修改映象名稱: ``` docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1  elasticsearch:latest ```  >ps[注意事項]： >1.名稱過長導致檢視些許不便，通過docker tag source-image[來源映象] target-image[目標映象]，推薦統一採用[target-image:target-version]格式定義，且不佔用空間，相當於重新命名映象 >2.對於拉取kibana[docker.elastic.co/kibana/kibana:7.3.1]和logstash[docker.elastic.co/logstash/logstash:7.3.1] 都建議修改。 4.部署映象服務： 部署命令： ``` docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest ``` 預設配置目錄： ``` /usr/share/elasticsearch/config /usr/share/elasticsearch/logs ``` 檢視容器列表： ``` docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}" ![docker ps] ``` (https://ftp.bmp.ovh/imgs/2019/11/d7e09058af3fadbd.png) ps[注意事項]: >1.需要開放埠[9200和9300]->9200作為Http協議，主要用於外部通訊,9300作為Tcp協議，jar之間就是通過tcp協議通訊,通常部署叢集就是通過9300通訊。推薦[宿主機自定義埠:9200] >2.--restart=always :配置容器重啟策略,當宿主機重啟由於配置了開機自啟動，不用手動啟動 >3.--privileged：配置容器操作許可權[true-root操作許可權，false-當前容器使用者操作許可權] >4.對於部署網路模式推薦預設橋接模式,也自定義可以host模式等 5.修改配置： 進入容器：docker exec -it container-id[容器id] or container-name[容器名稱] /bin/bash 例如：docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id  修改配置檔案: ```linux [root@f2d2e97da375 elasticsearch]# ls LICENSE.txt NOTICE.txt README.textile bin config data jdk lib logs modules plugins [root@f2d2e97da375 elasticsearch]# [root@f2d2e97da375 elasticsearch]# cd config [root@f2d2e97da375 config]# ls elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles [root@f2d2e97da375 config]# vi elasticsearch.yml ``` 新增跨域配置：`http.cors.enabled: true && http.cors.allow-origin: "*" ` ``` cluster.name: "docker-cluster" network.host: 0.0.0.0 http.cors.enabled: true http.cors.allow-origin: "*" ``` 然後退出exit容器，在宿主機重啟容器：docker restart container-id[容器id] or container-name[容器名稱] docker restart f2d2e97da375 ``` [root@f2d2e97da375 config]# exit exit [root@centos-meteor ~]# docker restart f2d2e97da375 f2d2e97da375 [root@centos-meteor ~]# ``` >ps[注意事項]： >1.進入容器方式：包括使用 docker attach 命令或 docker exec 命令， >推薦使用 docker exec 命令。原因： > * docker attach: 使用exit退出容器，會導致容器的停止 > * docker exec：使用exit退出容器，不會導致容器的停止 > * 參考docker進入容器的幾種方法部落格-[docker進入容器的幾種方法](https://www.cnblogs.com/yanshicheng/p/9452545.html) >2.如果Docker安裝了視覺化介面 Portainer，推薦採用這種方式進入容器： > ### 搭建部署ElasticSearch-Head服務 >ElasticSearch-Head：彈性搜尋叢集的Web前端介面，是使用Nodjs構建的，主要用於檢視ElasticSearch相關資訊 1.拉取Elasticsearch-Head 映象：docker pull mobz/elasticsearch-head:5 ``` [root@centos-amber ~]# docker pull mobz/elasticsearch-head:5 5: Pulling from mobz/elasticsearch-head 75a822cd7888: Pull complete 57de64c72267: Pull complete 4306be1e8943: Pull complete 871436ab7225: Pull complete 0110c26a367a: Pull complete 1f04fe713f1b: Pull complete 723bac39028e: Pull complete 7d8cb47f1c60: Pull complete 7328dcf65c42: Pull complete b451f2ccfb9a: Pull complete 304d5c28a4cf: Pull complete 4cf804850db1: Pull complete Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34 Status: Downloaded newer image for mobz/elasticsearch-head:5 docker.io/mobz/elasticsearch-head:5 [root@centos-amber ~]# ``` 2.修改Elasticsearch-Head 映象名稱：docker tag  mobz/elasticsearch-head:5   elasticsearch-head:latest ``` [root@centos-amber ~]# docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest [root@centos-amber ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE grafana/grafana latest 05d1bcf30d16 7 days ago 207MB nginx latest 540a289bab6c 3 weeks ago 126MB prom/prometheus latest 2c8e464e47f4 3 weeks ago 129MB moxm/sentinel-dashboard latest 0ccaac81584e 4 weeks ago 167MB portainer latest 4cda95efb0e4 4 weeks ago 80.6MB portainer/portainer latest 4cda95efb0e4 4 weeks ago 80.6MB apache/skywalking-ui latest fa66ca9c9862 2 months ago 123MB apache/skywalking-oap-server latest 376a37cdf65c 2 months ago 190MB docker.elastic.co/kibana/kibana 7.3.1 b54865ba6b0b 2 months ago 1.01GB docker.elastic.co/elasticsearch/elasticsearch 7.3.1 3d3aa92f641f 2 months ago 807MB elasticsearch latest 3d3aa92f641f 2 months ago 807MB prom/node-exporter latest e5a616e4b9cf 5 months ago 22.9MB google/cadvisor latest eb1210707573 12 months ago 69.6MB elasticsearch-head latest b19a5c98e43b 2 years ago 824MB mobz/elasticsearch-head 5 b19a5c98e43b 2 years ago 824MB tutum/influxdb latest c061e5808198 3 years ago 290MB [root@centos-amber ~]# ``` 3.部署Elasticsearch-Head 容器： ``` docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest ``` 檢視容器服務： ``` docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}" ```  4.瀏覽器訪問：http://remote-ip:9100/  ### 實戰測試ElasticSearch-Head服務 1.準備資料：  ```json { "companyId": "ebcb4f99e0cb4ad781278ae636c1031f", "classifyName": "激素檢測", "deviceType": "激素檢測", "data": { "deviceId": "20000060100000002", "deviceNo": "QC00020000060100000002", "appid": "00000000", "sim": "89860404191792655118", "csq": "30", "electric": "98", "voltage": "13279", "softVer": "5143000500010601-01020006|5143000500010601-01010003", "hardVer": "5143000500010601-00000002|5143000500010601-00000002", "status": "0", "date": "2020-09-13 11:23:52", "elements": [ { "key": "20", "value": "10", "time": "2020-09-13 11:23:52" }, { "key": "21", "value": "11", "time": "2020-09-13 11:23:52" }, { "key": "22", "value": "12", "time": "2020-09-13 11:23:52" }, { "key": "23", "value": "13", "time": "2020-09-13 11:23:52" } ] } } ``` 2.設定請求頭：application/json;charset=UTF-8  3.傳送PUT請求：http://remote-ip/device/deviceData/QC00020000060100000001  ```json { "_index": "device", "_type": "deviceData", "_id": "QC00020000060100000002", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": true } ``` 4.開啟瀏覽器檢視結果：  ### 實戰搭建ElasticSearch總結 1.部署基本命令： ```yml docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai" elasticsearch:latest ``` 2.配置檔案[elasticsearch.yml ]: ```yml cluster.name: elasticsearch-cluster node.name: elasticsearch-server network.host: 0.0.0.0 network.bind_host: 0.0.0.0 network.publish_host: 0.0.0.0 http.cors.enabled: true http.cors.allow-origin: "*" bootstrap.memory_lock: false bootstrap.system_call_filter: false transport.tcp.port: 9300 transport.tcp.compress: true http.max_content_length: 128mb ``` 3.分詞外掛安裝： ```yml ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip ``` 4.Docker 部署 ElasticSearch-Head： ``` docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest ``` 5.ElasticSearch-Head 使用406錯誤：  ajaxSettings部分json配置： ```json ajaxSettings: { url: ajaxLocation, isLocal: rlocalProtocol.test(ajaxLocParts[1]), global: true, type: "GET", //contentType: "application/x-www-form-urlencoded", contentType: "application/json;charset=UTF-8", processData: true, async: true, /* timeout: 0, data: null, dataType: null, username: null, password: null, cache: null, traditional: false, headers: {}, */ accepts: { xml: "application/xml, text/xml", html: "text/html", text: "text/plain", json: "application/json, text/javascript", "*": "*/*", }, ``` inspectData 部分json配置： ```js // Detect, normalize options and install callbacks for jsonp requests jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) { //var inspectData = s.contentType === "application/x-www-form-urlencoded" && var inspectData = (s.contentType === "application/x-www-form-urlencoded" && typeof s.data === "string") || (s.contentType === "application/json;charset=UTF-8" && typeof s.data === "string"); ``` > 版權宣告：本文為博主原創文章，遵循相關版權協議，如若轉載或者分享請附上原文出處連結和連結來源。