Prometheus-Prometheus-Opterator中新增監控etcd叢集
阿新 • • 發佈:2021-06-10
一、環境宣告
- kubeadm kubernetes 1.15
- etcd 也是叢集內 pod 部署方式,自帶metrics介面
- Prometheus-Operator
二、監控etcd叢集
2.1、檢視介面資訊
# https
# curl --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://127.0.0.1:2379/metrics -k
# http
# curl -L http://localhost:2379/metrics
2.2、檢視etcd叢集資訊獲取使用的證書資訊
# kubectl describe pods -n kube-system etcd-wt-rd-k8s-control-plane-01-beijing Name: etcd-wt-rd-k8s-control-plane-01-beijing Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: wt-rd-k8s-control-plane-01-beijing/10.2.3.141 Start Time: Thu, 27 May 2021 07:19:25 +0000 Labels: component=etcd tier=control-plane Annotations: kubernetes.io/config.hash: 2c510faa262b7e6cc922f5c10917a5a4 kubernetes.io/config.mirror: 2c510faa262b7e6cc922f5c10917a5a4 kubernetes.io/config.seen: 2019-09-03T07:15:31.882345426Z kubernetes.io/config.source: file Status: Running IP: 10.2.3.141 Containers: etcd: Container ID: docker://7c0fece5de2b5ea89b5b648bebf2f076320d379500ee2f677dd0619963449bc5 Image: k8s.gcr.io/etcd:3.3.10 Image ID: docker://sha256:2c4adeb21b4ff8ed3309d0e42b6b4ae39872399f7b37e0856e673b13c4aba13d Port: <none> Host Port: <none> Command: etcd --advertise-client-urls=https://10.2.3.141:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://10.2.3.141:2380 --initial-cluster=wt-rd-k8s-control-plane-01-beijing=https://10.2.3.141:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://10.2.3.141:2379 --listen-peer-urls=https://10.2.3.141:2380 --name=wt-rd-k8s-control-plane-01-beijing --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
2.3、利用 kubectl 命令將三個證書檔案存入 Kubernetes 的 Secret 資源下
- 可以看出etcd使用的證書都在對應節點的/etc/kubernetes/pki/etcd/目錄下面。所以先將需要使用的證書通過secret物件儲存到叢集中:
# kubectl -n monitoring create secret generic etcd-certs \ --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt \ --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key \ --from-file=/etc/kubernetes/pki/etcd/ca.crt
2.4、建立etcd-certs物件配置到prometheus資源物件
# kubectl edit prometheus k8s -n monitoring # 新增secrets的如下屬性: nodeSelector: kubernetes.io/os: linux podMonitorSelector: {} replicas: 2 # 新增如下兩行 secrets: - etcd-certs # 更新完成後,就可以在Prometheus的Pod中獲取之前建立的etcd證書檔案了。先檢視一下pod名字 kubectl get po -n monitoring NAME READY STATUS RESTARTS AGE ... prometheus-k8s-0 3/3 Running 1 2m20s prometheus-k8s-1 3/3 Running 1 3m19s ... # 進入兩個容器,檢視一下證書的具體路徑 kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod. /prometheus $ ls /etc/prometheus/secrets/etcd-certs/ ca.crt healthcheck-client.crt healthcheck-client.key
2.5、建立ServiceMonitor
MonitorEtcd# cat prometheus-serviceMonitorEtcd.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: etcd-k8s
namespace: monitoring
labels:
k8s-app: etcd-k8s
spec:
jobLabel: k8s-app
endpoints:
- port: port
interval: 15s
scheme: https
tlsConfig:
caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
insecureSkipVerify: true
selector:
matchLabels:
k8s-app: etcd
namespaceSelector:
matchNames:
- kube-system
# kubectl apply -f prometheus-serviceMonitorEtcd.yaml
servicemonitor.monitoring.coreos.com/etcd-k8s created
2.6、建立Service
- ServiceMonitor已經建立完成了,需要建立一個對應的Service物件。prometheus-etcdService.yaml內容如下:
MonitorEtcd# cat prometheus-etcdService.yaml
apiVersion: v1
kind: Service
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
spec:
type: ClusterIP
clusterIP: None #設定為None,不分配Service IP
ports:
- name: port
port: 2379
---
apiVersion: v1
kind: Endpoints
metadata:
name: etcd-k8s
namespace: kube-system
labels:
k8s-app: etcd
subsets:
- addresses:
- ip: 10.2.3.141 # 指定etcd節點地址,如果是叢集則繼續向下新增
- ip: 10.2.3.179
- ip: 10.2.4.121
ports:
- name: port
port: 2379 # ETCD埠
protocol: TCP
# etcd叢集獨立於叢集之外,所以需要定義一個Endpoints。Endpoints的metadata區域的內容要和Service保持一致,並且將Service的clusterIP設定為None。
# 在Endpoints的subsets中填寫etcd的地址,如果是叢集,則在addresses屬性下面新增多個地址。
# kubectl apply -f prometheus-etcdService.yaml
service/etcd-k8s created
endpoints/etcd-k8s created
三、檢視 Prometheus 規則及Grafana 引入 ETCD 儀表盤
3.1、檢視 Prometheus 規則
3.2、Grafana 引入 ETCD 儀表盤
- 開啟官網來的如下圖所示,點選下載JSO檔案
- grafana官網:https://grafana.com/grafana/dashboards/3070
- 中文版ETCD叢集外掛:https://grafana.com/grafana/dashboards/9733