一、監控使用exporter暴露metrics資訊的服務

　　對於一些應用如：kafka、redis或者mysql等，其需要使用exporter來暴露本身的metrics資訊。這些服務的監控，可以使用prometheus operator中的servicemonitor來匹配該服務的exporter的service，達到採集metrics的目的。

　　現對kafka進行資料採集和監控

　　kafka_exporter地址：https://github.com/danielqsj/kafka_exporter

　　安裝kafka_exporter

　　kafka-exporter.yaml　

apiVersion: v1
kind: Service
metadata:
  name: kafka
 
-exporter
  namespace: monitoring
  labels:
    k8s-app: kafka-exporter
spec:
  selector:
    k8s-app: kafka-exporter
  ports:
  - name: kafka-exporter
    port: 9308
    protocol: TCP
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-exporter
  namespace: monitoring
  labels:
    k8s
 
-app: kafka-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kafka-exporter
  template:
    metadata:
      labels:
        k8s-app: kafka-exporter
    spec:
      containers:
      - name: kafka-exporter
        image: danielqsj/kafka-exporter:latest
        imagePullPolicy: IfNotPresent
        args: [
 
"--kafka.server=kafka-0.kafka-headless.public-service:9092"]
        resources:
          requests:
            memory: 100M
        volumeMounts:
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
      nodeSelector:
        monitor: "true"
      volumes:
        - name: tz-config
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai

• 建立了kafka_exporter的service和deployment，service用於和Prometheus servicemonitor進行匹配，採集資料
• args：Kafka_exporter啟動引數，為k8s叢集中kafka的地址

檢視

# kubectl get svc,deploy -n monitoring -l k8s-app=kafka-exporter
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kafka-exporter   ClusterIP   10.98.228.115   <none>        9308/TCP   50m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafka-exporter   1/1     1            1           49m

測試

# curl 10.98.228.115:9308/metrics | more


# TYPE go_gc_duration_seconds summary
1086k  go_gc_duration_seconds{quantile="0"} 7.7719e-05
 go_gc_duration_seconds{quantile="0.25"} 0.00010556
  go_gc_duration_seconds{quantile="0.5"} 0.0001406
 0 go_gc_duration_seconds{quantile="0.75"} 0.00019177
--go_gc_duration_seconds{quantile="1"} 0.001047939
:-go_gc_duration_seconds_sum 0.004194563
-:go_gc_duration_seconds_count 21
--# HELP go_goroutines Number of goroutines that currently exist.
 -# TYPE go_goroutines gauge
-go_goroutines 11
...

• kafka中有topic時，exporter才能採集更多的資料

二、配置servicemonitor

　　prometheus-serviceMonitorKafka.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: kafka-exporter
    scheme: http
    tlsConfig:
      insecureSkipVerify: false
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      k8s-app: kafka-exporter

• 該servicemonitor匹配的是 monitoring這個namespace下，label為k8s-app=kafka-exporter這個service

檢視Prometheus