ElasticSearch + SearchGuard 叢集搭建
阿新 • • 發佈：2019-12-31
背景介紹
業務原有ES叢集因證書到期，需要進行證書更換操作。由於更新證書後需重啟整個集群后才能生效，且業務敏感度較高，無法容忍整個叢集停止對外服務。
解決方案
-
OP負責部署一套新ES叢集，版本配置同原叢集一致 -
RD負責開發程式- 將歷史資料匯入新叢集
- 業務服務實現雙寫兩套
ES叢集功能
-
兩套叢集資料一致後，
OP和RD配合切換業務流量 -
OP將原有叢集更新證書
ElasticSearch 叢集
準備
-
ES版本 5.5.1 - Search Guard
-
JAVA版本 jdk1.8.0_161
叢集配置
3臺 Master Node (56核/128G/3.7T普通硬碟)
10.90.104.133
10.90.105.133
10.90.106.133
5臺 Data Node (56核/128G/3.7T普通硬碟)
10.90.107.132
10.90.108.132
10.90.109.133
10.90.110.133
10.90.111.133
埠: 9201/9301
複製程式碼 系統調整
臨時生效
$ sudo sysctl -w vm.max_map_count=262144
永久生效
$ grep vm.max_map_count /etc/sysctl.conf
$ echo vm.max_map_count=262144 >> /etc/sysctl.conf
or
$ vi /etc/sysctl.conf
vm.max_map_count=262144
複製程式碼安裝JAVA
下載 jdk1.8.0_161(若連結404，自行註冊賬號下載)
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" 安裝ElasticSearch
建立使用者
$ adduser -U -s /sbin/nologin elasticsearch
建立目錄
$ mkdir -p /usr/local/elasticsearch
$ mkdir -p /data/elasticsearch/my-project
$ mkdir -p /data/logs/elasticsearch/my-project/{log,pid}
下載es 5.5.1
$ cd /usr/local/elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz
解壓檔案
$ tar zxf elasticsearch-5.5.1.tar.gz && rm -rf elasticsearch-5.5.1.tar.gz
$ mv elasticsearch-5.5.1 my-project-es
複製程式碼安裝Search Guard外掛
下載外掛
$ cd {PROJECT_NAME}
$ wget http://search.maven.org/remotecontent?filepath=com/floragunn/search-guard-5/5.5.1-15/search-guard-5-5.5.1-15.zip
安裝外掛
$ bin/elasticsearch-plugin install search-guard-5-5.5.1-15.zip
複製程式碼生成證書
下載工具(路徑沒有特殊要求)
$ git clone https://github.com/floragunncom/search-guard-ssl.git
生成證書
$ cd search-guard-ssl/example-pki-scripts/
$ ./clean.sh
$ ./gen_root_ca.sh kubernetes- kubernetes- # COMMAND <CA_PASS> <TS_PASS>
$ ./gen_node_cert.sh k8s-es-node kubernetes- kubernetes- # COMMAND <NODE_NAME> <KS_PASS> <CA_PASS>
$ ./gen_client_node_cert.sh sgadmin kubernetes- kubernetes- # COMMAND <CLIENT_NAME> <KS_PASS> <CA_PASS>
拷貝證書到 config 目錄下
$ cp *jks /usr/local/elasticsearch/my-project/config/
複製程式碼配置ElasticSearch
備份配置
$ cd /usr/local/elasticsearch/my-project/config/
$ mv elasticsearch.yml{,bak}
Node Master (三臺master都按此配置):
主配置: 不做儲存，不開啟 http
$ vim elasticsearch.yml
cluster.name: my-project-es
node.name: 10.90.104.133
node.master: true
node.data: false
node.ingest: false
path:
data:
- /data/elasticsearch/my-project
logs: /data/logs/elasticsearch/my-project/log
thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000
bootstrap.memory_lock: true
network.host: 10.90.104.133
http.port: 9201
transport.tcp.port: 9301
http:
enabled: false
compression: true
cors:
enabled: true
allow-origin: "*"
allow-headers: Authorization
discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90.105.133:9301,10.90.106.133:9301]
discovery.zen.minimum_master_nodes: 2
searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks
searchguard.ssl.transport.keystore_password: kubernetes-
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: kubernetes-
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE
記憶體配置:
$ vim jvm.options
...
Xms20g
Xmx20g
...
Node Data (後5臺資料節點都按此配置):
主配置: 儲存資料並開啟 http 訪問
$ vim elasticsearch.yml
cluster.name: my-project-es
node.name: 10.90.107.132
node.master: false
node.data: true
node.ingest: true
path:
data:
- /data/elasticsearch/my-project
logs: /data/logs/elasticsearch/my-project/log
thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000
bootstrap.memory_lock: true
network.host: 10.90.107.132
http.port: 9201
transport.tcp.port: 9301
http:
enabled: true
compression: true
cors:
enabled: true
allow-origin: "*"
allow-headers: Authorization
discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,C=DE
記憶體配置:
$ vim jvm.options
...
Xms63g
Xmx63g
...
複製程式碼使用者許可權及密碼
$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5
生成密碼
$ sh tools/hash.sh -p ICdsy1B3j68Tr4w2
$2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
修改使用者預設密碼,新增使用者 myuser
$ vim sgconfig/sg_internal_users.yml
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
admin:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
logstash:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
kibanaserver:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
kibanaro:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
roles:
- kibanarole
readall:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
myuser:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
# password is: ICdsy1B3j68Tr4w2
修改角色對映檔案(分配admin許可權)
$ vim sgconfig/sg_roles_mapping.yml
...
# 僅將新使用者新增到該許可權下，其他配置保持預設
sg_all_access:
users:
- admin
- myuser
...
複製程式碼新增服務管理指令碼
$ vim /usr/lib/systemd/system/es-my-project.service
[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
Environment=ES_HOME=/usr/local/elasticsearch/my-project
Environment=CONF_DIR=/usr/local/elasticsearch/my-project/config
Environment=PID_DIR=/data/logs/elasticsearch/my-project/pid
WorkingDirectory=/usr/local/elasticsearch/my-project
User=elasticsearch
Group=elasticsearch
ExecStartPre=/usr/local/elasticsearch/my-project/bin/elasticsearch-systemd-pre-exec
ExecStart=/usr/local/elasticsearch/my-project/bin/elasticsearch \
-p \${PID_DIR}/elasticsearch.pid \
-Edefault.path.conf=\${CONF_DIR}
StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
LimitNPROC=2048
LimitMEMLOCK=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
複製程式碼修改使用者
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch
chown -R elasticsearch:elasticsearch /data/logs/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
chown -R elasticsearch:elasticsearch /usr/lib/systemd/system/es-my-project.service
複製程式碼啟動服務
新增開機自啟
sudo systemctl daemon-reload
sudo systemctl enable es-my-project.service
sudo systemctl start es-my-project
複製程式碼sguard 初始化密碼
sguard配置生效，只需要在叢集中的任意節點執行即可(推薦固定一臺master操作)
更改sgconfig配置就需要執行本指令碼，執行過程中不會重啟ES叢集。
$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5/tools
$ ./sgadmin.sh --hostname 10.90.104.133 --port 9301 \
-cd ../sgconfig/ \
-ks /usr/local/elasticsearch/my-project/config/sgadmin-keystore.jks \
-kspass kubernetes- \
-ts ./usr/local/elasticsearch/my-project/config/truststore.jks \
-tspass kubernetes- \
-nhnv --diagnose -cn my-project-es
複製程式碼檢視叢集狀態
$ curl -u myuser:ICdsy1B3j68Tr4w2 -XGET http://10.90.107.132:9201/_cluster/health?pretty
{
"cluster_name" : "my-project-es","status" : "green","timed_out" : false,"number_of_nodes" : 3,"number_of_data_nodes" : 3,"active_primary_shards" : 4,"active_shards" : 9,"relocating_shards" : 0,"initializing_shards" : 0,"unassigned_shards" : 0,"delayed_unassigned_shards" : 0,"number_of_pending_tasks" : 0,"number_of_in_flight_fetch" : 0,"task_max_waiting_in_queue_millis" : 0,"active_shards_percent_as_number" : 100.0
}
複製程式碼Kibana 部署
Dockerfile
FROM docker.xxx.com/library/kibana-searchguard:5.5.1
ENV CLUSTER_NAME=my-project-es \
SERVER_BASEPATH= \
ELASTICSEARCH_URL="http://10.90.107.132:9201" \
XPACK_SECURITY_ENABLED=false \
XPACK_GRAPH_ENABLED=false \
XPACK_ML_ENABLED=false \
ELASTICSEARCH_USERNAME=kibanaserver \
ELASTICSEARCH_PASSWORD=ICdsy1B3j68Tr4w2
# port is 5601
複製程式碼基於內部倉庫映象構建
構建映象+生成容器
$ docker build -t my-project-kibana .
$ docker run -d my-project-kibana
複製程式碼登入 kibana
進行初始化設定
更換證書
新生成的證書檔案*.jks拷貝到原叢集config下，重啟ES叢集和search guard即可。