Filebeat 收集K8S 日誌,生產環境實踐
阿新 • • 發佈:2020-08-11
根據生產環境要求,需要採集K8S Pod 日誌,和開發協商之後,Pod中應用會將日誌輸出到容器終端上,這時可以直接用filebeat 採集node節點上面的/var/log/containers/*.log
日誌,然後將日誌輸出到kafka訊息佇列中,經過kafka將日誌寫入logstash進行格式化,然後由logstash傳入elasticsearch儲存,然後kibana會連線elasticsearch展示索引資料。
資料傳輸流程:Pod -> /var/log/containers/*.log
-> Filebeat -> Kafka叢集 -> Logstash -> Elasticsearch -> Kibana
K8S 配置Filebeat
整體配置檔案如下:
$ ls
filebeat.daemonset.yml filebeat.permission.yml
filebeat.indice-lifecycle.configmap.yml filebeat.settings.configmap.yml
Filebeat操作許可權
$ cat filebeat.permission.yml --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: filebeat labels: app: filebeat rules: - apiGroups: [""] resources: - namespaces - pods verbs: - get - watch - list --- apiVersion: v1 kind: ServiceAccount metadata: namespace: kube-system name: filebeat labels: app: filebeat
Filebeat主配置檔案
注意:如果收集Java堆疊錯誤日誌,需要增加下面帶註釋的幾行引數,multiline多行處理解決次問題。
$ cat filebeat.settings.configmap.yml --- apiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: filebeat-config labels: app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: container enabled: true paths: - /var/log/containers/*.log multiline: # 多行處理,正則表示如果前面幾個數字不是4個數字開頭,那麼就會合併到一行,解決Java堆疊錯誤日誌收集問題 pattern: ^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2} #匹配Java日誌開頭時間 negate: true # 正則是否開啟,預設false不開啟 match: after # 不匹配的正則的行是放在上面一行的前面還是後面 processors: - add_kubernetes_metadata: in_cluster: true host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" - add_cloud_metadata: - add_kubernetes_metadata: matchers: - logs_path: logs_path: "/var/log/containers/" - add_docker_metadata: output: kafka: enabled: true # 增加kafka的輸出 hosts: ["10.0.0.72:9092"] topic: filebeat max_message_bytes: 5242880 partition.round_robin: reachable_only: true keep-alive: 120 required_acks: 1 setup.ilm: policy_file: /etc/indice-lifecycle.json
Filebeat索引生命週期策略配置
ElasticSearch 的 indice 生命週期表示一組規則,可以根據 indice 的大小或者時長應用到你的 indice 上。比如可以每天或者每次超過 1GB 大小的時候對 indice 進行輪轉,我們也可以根據規則配置不同的階段。由於監控會產生大量的資料,很有可能一天就超過幾十G的資料,所以為了防止大量的資料儲存,我們可以利用 indice 的生命週期來配置資料保留,這個在 Prometheus 中也有類似的操作。 如下所示的檔案中,我們配置成每天或每次超過5GB的時候就對 indice 進行輪轉,並刪除所有超過30天的 indice 檔案,我們這裡只保留30天監控資料完全足夠了。
filebeat.indice-lifecycle.configmap.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: kube-system
name: filebeat-indice-lifecycle
labels:
app: filebeat
data:
indice-lifecycle.json: |-
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "5GB" ,
"max_age": "1d"
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
Filebeat Daemonset配置檔案
$ cat filebeat.daemonset.yml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
namespace: kube-system
name: filebeat
labels:
app: filebeat
spec:
selector:
matchLabels:
app: filebeat
template:
metadata:
labels:
app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.8.0
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: filebeat-indice-lifecycle
mountPath: /etc/indice-lifecycle.json
readOnly: true
subPath: indice-lifecycle.json
- name: data
mountPath: /usr/share/filebeat/data
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: dockersock
mountPath: /var/run/docker.sock
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: filebeat-indice-lifecycle
configMap:
defaultMode: 0600
name: filebeat-indice-lifecycle
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
執行到K8S中
$ kubectl apply -f filebeat.settings.configmap.yml \
-f filebeat.indice-lifecycle.configmap.yml \
-f filebeat.daemonset.yml \
-f filebeat.permissions.yml
configmap/filebeat-config created
configmap/filebeat-indice-lifecycle created
daemonset.apps/filebeat created
clusterrolebinding.rbac.authorization.k8s.io/filebeat created
clusterrole.rbac.authorization.k8s.io/filebeat created
serviceaccount/filebeat created