kubernetes -- helm charts 開發: 2、 編寫daemonset
1 概要介紹
DaemonSet:
特點：
確保全部Node上執行一個Pod副本（單副本）
Node加入叢集，則新增Pod；移除，則回收
刪除Daemonset,刪除建立的Pod
2編寫
2.1 必須欄位
apiVersion:
可用值： extensions/v1beta1
kind :
可用值：DaemonSet
metadata:
例如：name: nova-compute
.spec:
唯一必須的欄位：
.spec.template
例如：
metadata:
labels:
{{ tuple $envAll .Chart.Name $daemonset | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
annotations:
configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}
.spec.template：
含義：是Pod模板。Pod有明確的格式。Pod中必須指明標籤合重啟策略，預設always。
具體參見：
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#pod-template
2.2 Pod Selector
.spec.selector：
表示Pod Selector.是物件，包含:
matchLabels:
matchEx[ressions:
終於發現不能執行的原因：
zabbix-damonset:
{{ tuple $envAll "zabbix_agent" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}
nova-daemonset:
{{ tuple $envAll $daemonset | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}
ceilometer現在寫的是：
{{ tuple $envAll | include "helm-toolkit.snippets.kubernetes_upgrades_deployment" | indent 2 }}
如果指定spec.selector必須和.spec.template.metadata.labels
匹配，目前沒有發現spec.selector
2.3 .spec.template.spec.nodeSelector
如果指定.spec.template.spec.nodeSelector，則Daemonset Controller將在能夠匹配上Node Selector的Node上建立Pod,在能夠匹配上的Node Affinity的Node上建立Pod。
目前nova:
nodeSelector:
{{ .Values.labels.agent.compute.node_selector_key }}: {{ .Values.labels.agent.compute.node_selector_value }}
沒有affinity，所以會在所有Node上建立Pod
ceilometer:
nodeSelector:
{{ .Values.labels.compute.node_selector_key }}: {{ .Values.labels.compute.node_selector_value }}
affinity:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_pod_anti_affinity" | indent 8 }}
3 如何排程Daemon Pod
Pod執行在哪個機器是由Kubernetes排程器進行選擇的，Daemon Controller建立的Pod已經確定在哪個機器上。
4 與Daemon Pod通訊
模式:
Push: Pod向其他Service傳送更新
NodeIP和的埠：使用hostPort,可通過Node IP訪問到Pod
客戶端知道Node IP列表
DNS:建立相同Pod Selector的Headless Service.通過endpoints資源檢索
Service:建立具有相同Pod Selector的Service,用該Service訪問到隨機Node上的daemon
5 更新DaemonSet
修改Node Label,DaemonSet向新匹配的Node新增Pod
6 init指令碼
通過直接在Node上啟動daemon程序。
7 一個ceilometer-compute的daemonset例子
7.1修改檔案統計
values.yaml
daemonset-compute.yaml
7.2 values.yaml
manifest部分:
daemonset_compute: true
deployment_notification: true
lifecycle部分:
新增：
daemonsets:
pod_replacement_strategy: RollingUpdate
compute:
enabled: true
min_ready_seconds: 0
max_unavailable: 1
replicas部分：
刪除,容易遺漏,因為daemonset沒有副本數的概念，deployment才有
compute: 3
7.3 daemonset-compute.yaml
判斷部分：
{{- if .Values.manifests.daemonset_compute }}
appversion：
apiVersion: extensions/v1beta1
kind：
kind: DaemonSet
affinity：
刪除（其他都沒有）
daemonset模板
{{ tuple $envAll "compute" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}
7.4 最終的ceilometer-compute的daemonset樣例
{{- if .Values.manifests.daemonset_compute }}
{{- $envAll := . }}
{{- $dependencies := .Values.dependencies.compute }}
{{- $mounts_ceilometer_compute := .Values.pod.mounts.ceilometer_compute.ceilometer_compute }}
{{- $mounts_ceilometer_compute_init := .Values.pod.mounts.ceilometer_compute.init_container }}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ceilometer-compute
spec:
{{ tuple $envAll "compute" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}
selector:
matchLabels:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 6 }}
template:
metadata:
labels:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
annotations:
configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}
configmap-etc-hash: {{ tuple "configmap-etc.yaml" . | include "helm-toolkit.utils.hash" }}
spec:
hostNetwork: true
hostPID: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
{{ .Values.labels.compute.node_selector_key }}: {{ .Values.labels.compute.node_selector_value }}
initContainers:
{{ tuple $envAll $dependencies $mounts_ceilometer_compute_init | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
containers:
- name: ceilometer-compute
image: {{ .Values.images.tags.ceilometer_compute }}
imagePullPolicy: {{ .Values.images.pull_policy }}
{{ tuple $envAll $envAll.Values.pod.resources.compute | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
command:
- /tmp/ceilometer-compute.sh
volumeMounts:
- name: pod-etc-ceilometer
mountPath: /etc/ceilometer
- name: ceilometer-etc
mountPath: /etc/ceilometer/ceilometer.conf
subPath: ceilometer.conf
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/api_paste.ini
subPath: api_paste.ini
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/policy.json
subPath: policy.json
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/event_definitions.yaml
subPath: event_definitions.yaml
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/event_pipeline.yaml
subPath: event_pipeline.yaml
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/pipeline.yaml
subPath: pipeline.yaml
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/gnocchi_resources.yaml
subPath: gnocchi_resources.yaml
readOnly: true
- name: ceilometer-etc
mountPath: /etc/ceilometer/mapping.json
subPath: mapping.json
readOnly: true
- name: ceilometer-bin
mountPath: /tmp/ceilometer-compute.sh
subPath: ceilometer-compute.sh
readOnly: true
- name: varlibnova
mountPath: /var/lib/nova
- name: varliblibvirt
mountPath: /var/lib/libvirt
- name: run
mountPath: /run
- name: cgroup
mountPath: /sys/fs/cgroup
- name: machine-id
mountPath: /etc/machine-id
readOnly: true
{{ if $mounts_ceilometer_compute.volumeMounts }}{{ toYaml $mounts_ceilometer_compute.volumeMounts | indent 12 }}{{ end }}
volumes:
- name: pod-etc-ceilometer
emptyDir: {}
- name: ceilometer-etc
configMap:
name: ceilometer-etc
defaultMode: 0444
- name: ceilometer-bin
configMap:
name: ceilometer-bin
defaultMode: 0555
- name: varlibnova
hostPath:
path: /var/lib/nova
- name: varliblibvirt
hostPath:
path: /var/lib/libvirt
- name: run
hostPath:
path: /run
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: machine-id
hostPath:
path: /etc/machine-id
{{ if $mounts_ceilometer_compute.volumes }}{{ toYaml $mounts_ceilometer_compute.volumes | indent 8 }}{{ end }}
{{- end }}
8 總結
daemonset相比較於deployment的最大區別就是當有新的節點加入進來的時候，daemonset會自動新建該daemonset的pod，這種場景往往適用於需要從每個新加入的節點獲取某種資訊。典型的例子就是: nova-compute, ceilometer-compute,因為當新增計算節點的時候，nova-compute和ceilometer-compute需要從新新增的計算節點上獲取虛機資訊。
參考:
[1] https://jimmysong.io/kubernetes-handbook/concepts/daemonset.html