杜楊浩，騰訊雲高階工程師，熱衷於開源、容器和Kubernetes。目前主要從事映象倉庫、Kubernetes叢集高可用&備份還原，以及邊緣計算相關研發工作。 ## 前言 SuperEdge分散式健康檢查功能由邊端的edge-health-daemon以及雲端的edge-health-admission組成： - edge-health-daemon：對同區域邊緣節點執行分散式健康檢查，並向apiserver傳送健康狀態投票結果(給node打annotation) - edge-health-admission：不斷根據node edge-health annotation調整kube-controller-manager設定的node taint(去掉NoExecute taint)以及endpoints(將失聯節點上的pods從endpoint subsets notReadyAddresses移到addresses中)，從而實現雲端和邊端共同決定節點狀態 整體架構如下所示：  之所以建立edge-health-admission雲端元件，是因為當雲邊斷連時，kube-controller-manager會執行如下操作： - 失聯的節點被置為ConditionUnknown狀態，並被新增NoSchedule和NoExecute的taints - 失聯的節點上的pod從Service的Endpoint列表中移除 當edge-health-daemon在邊端根據健康檢查判斷節點狀態正常時，會更新node：去掉NoExecute taint。但是在node成功更新之後又會被kube-controller-manager給刷回去(再次新增NoExecute taint)，因此必須新增Kubernetes mutating admission webhook也即edge-health-admission，將kube-controller-manager對node api resource的更改做調整，最終實現分散式健康檢查效果 在深入原始碼之前先介紹一下[Kubernetes Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) > > An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized. The controllers consist of the list below, are compiled into the kube-apiserver binary, and may only be configured by the cluster administrator. In that list, there are two special controllers: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These execute the mutating and validating (respectively) admission control webhooks which are configured in the API. Kubernetes Admission Controllers是kube-apiserver處理api請求的某個環節，用於在api請求認證&鑑權之後，物件持久化之前進行呼叫，對請求進行校驗或者修改(or both) Kubernetes Admission Controllers包括多種admission，大多數都內嵌在kube-apiserver程式碼中了。其中MutatingAdmissionWebhook以及ValidatingAdmissionWebhook controller比較特殊，它們分別會呼叫外部構造的mutating admission control webhooks以及validating admission control webhooks > > Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies. Admission Webhooks是一個HTTP回撥服務，接受AdmissionReview請求並進行處理，按照處理方式的不同，可以將Admission Webhooks分類如下： - [validating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)：通過ValidatingWebhookConfiguration配置，會對api請求進行准入校驗，但是不能修改請求物件 - [mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook)：通過MutatingWebhookConfiguration配置，會對api請求進行准入校驗以及修改請求物件 兩種型別的webhooks都需要定義如下Matching requests欄位： - admissionReviewVersions：定義了apiserver所支援的AdmissionReview api resoure的版本列表(API servers send the first AdmissionReview version in the admissionReviewVersions list they support) - name：webhook名稱(如果一個WebhookConfiguration中定義了多個webhooks，需要保證名稱的唯一性) - clientConfig：定義了webhook server的訪問地址(url or service)以及CA bundle(optionally include a custom CA bundle to use to verify the TLS connection) - namespaceSelector：限定了匹配請求資源的名稱空間labelSelector - objectSelector：限定了匹配請求資源本身的labelSelector - rules：限定了匹配請求的operations，apiGroups，apiVersions，resources以及resource scope，如下： - operations：規定了請求操作列表(Can be "CREATE", "UPDATE", "DELETE", "CONNECT", or "*" to match all.) - apiGroups：規定了請求資源的API groups列表("" is the core API group. "*" matches all API groups.) - apiVersions：規定了請求資源的API versions列表("*" matches all API versions.) - resources：規定了請求資源型別(node, deployment and etc) - scope：規定了請求資源的範圍(Cluster，Namespaced or *) - timeoutSeconds：規定了webhook迴應的超時時間，如果超時了，根據failurePolicy進行處理 - failurePolicy：規定了apiserver對admission webhook請求失敗的處理策略： - Ignore：means that an error calling the webhook is ignored and the API request is allowed to continue. - Fail：means that an error calling the webhook causes the admission to fail and the API request to be rejected. - matchPolicy：規定了rules如何匹配到來的api請求，如下： - Exact：完全匹配rules列表限制 - Equivalent：如果修改請求資源(apiserver可以實現物件在不同版本的轉化)可以轉化為能夠配置rules列表限制，則認為該請求匹配，可以傳送給admission webhook - reinvocationPolicy：In v1.15+, to allow mutating admission plugins to observe changes made by other plugins, built-in mutating admission plugins are re-run if a mutating webhook modifies an object, and mutating webhooks can specify a reinvocationPolicy to control whether they are reinvoked as well. - Never: the webhook must not be called more than once in a single admission evaluation - IfNeeded: the webhook may be called again as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call. - Side effects：某些webhooks除了修改AdmissionReview的內容外，還會連帶修改其它的資源("side effects")。而sideEffects指示了Webhooks是否具有"side effects"，取值如下： - None: calling the webhook will have no side effects. - NoneOnDryRun: calling the webhook will possibly have side effects, but if a request with dryRun: true is sent to the webhook, the webhook will suppress the side effects (the webhook is dryRun-aware). 這裡給出edge-health-admission對應的MutatingWebhookConfiguration作為參考示例： ``` apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: edge-health-admission webhooks: - admissionReviewVersions: - v1 clientConfig: caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNwRENDQVl3Q0NRQ2RaL0w2akZSSkdqQU5CZ2txaGtpRzl3MEJBUXNGQURBVU1SSXdFQVlEVlFRRERBbFgKYVhObE1tTWdRMEV3SGhjTk1qQXdOekU0TURRek9ERTNXaGNOTkRjeE1qQTBNRFF6T0RFM1dqQVVNUkl3RUFZRApWUVFEREFsWGFYTmxNbU1nUTBFd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUNSCnhHT2hrODlvVkRHZklyVDBrYVkwajdJQVJGZ2NlVVFmVldSZVhVcjh5eEVOQkF6ZnJNVVZyOWlCNmEwR2VFL3cKZzdVdW8vQWtwUEgrbzNQNjFxdWYrTkg1UDBEWHBUd1pmWU56VWtyaUVja3FOSkYzL2liV0o1WGpFZUZSZWpidgpST1V1VEZabmNWOVRaeTJISVF2UzhTRzRBTWJHVmptQXlDMStLODBKdDI3QUl4YmdndmVVTW8xWFNHYnRxOXlJCmM3Zk1QTXJMSHhaOUl5aTZla3BwMnJrNVdpeU5YbXZhSVA4SmZMaEdnTU56YlJaS1RtL0ZKdDdyV0dhQ1orNXgKV0kxRGJYQ2MyWWhmbThqU1BqZ3NNQTlaNURONDU5ellJSkVhSTFHeFI3MlhaUVFMTm8zdE5jd3IzVlQxVlpiTgo1cmhHQlVaTFlrMERtd25vWTBCekFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUhuUDJibnJBcWlWCjYzWkpMVzM0UWFDMnRreVFScTNVSUtWR3RVZHFobWRVQ0I1SXRoSUlleUdVRVdqVExpc3BDQzVZRHh4YVdrQjUKTUxTYTlUY0s3SkNOdkdJQUdQSDlILzRaeXRIRW10aFhiR1hJQ3FEVUVmSUVwVy9ObUgvcnBPQUxhYlRvSUVzeQpVNWZPUy9PVVZUM3ZoSldlRjdPblpIOWpnYk1SZG9zVElhaHdQdTEzZEtZMi8zcEtxRW1Cd1JkbXBvTExGbW9MCmVTUFQ4SjREZExGRkh2QWJKalFVbjhKQTZjOHUrMzZJZDIrWE1sTGRZYTdnTnhvZTExQTl6eFJQczRXdlpiMnQKUXZpbHZTbkFWb0ZUSVozSlpjRXVWQXllNFNRY1dKc3FLMlM0UER1VkNFdlg0SmRCRlA2NFhvU08zM3pXaWhtLworMXg3OXZHMUpFcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= service: namespace: kube-system name: edge-health-admission path: /node-taint failurePolicy: Ignore matchPolicy: Exact name: node-taint.k8s.io namespaceSelector: {} objectSelector: {} reinvocationPolicy: Never rules: - apiGroups: - '*' apiVersions: - '*' operations: - UPDATE resources: - nodes scope: '*' sideEffects: None timeoutSeconds: 5 - admissionReviewVersions: - v1 clientConfig: caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNwRENDQVl3Q0NRQ2RaL0w2akZSSkdqQU5CZ2txaGtpRzl3MEJBUXNGQURBVU1SSXdFQVlEVlFRRERBbFgKYVhObE1tTWdRMEV3SGhjTk1qQXdOekU0TURRek9ERTNXaGNOTkRjeE1qQTBNRFF6T0RFM1dqQVVNUkl3RUFZRApWUVFEREFsWGFYTmxNbU1nUTBFd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUNSCnhHT2hrODlvVkRHZklyVDBrYVkwajdJQVJGZ2NlVVFmVldSZVhVcjh5eEVOQkF6ZnJNVVZyOWlCNmEwR2VFL3cKZzdVdW8vQWtwUEgrbzNQNjFxdWYrTkg1UDBEWHBUd1pmWU56VWtyaUVja3FOSkYzL2liV0o1WGpFZUZSZWpidgpST1V1VEZabmNWOVRaeTJISVF2UzhTRzRBTWJHVmptQXlDMStLODBKdDI3QUl4YmdndmVVTW8xWFNHYnRxOXlJCmM3Zk1QTXJMSHhaOUl5aTZla3BwMnJrNVdpeU5YbXZhSVA4SmZMaEdnTU56YlJaS1RtL0ZKdDdyV0dhQ1orNXgKV0kxRGJYQ2MyWWhmbThqU1BqZ3NNQTlaNURONDU5ellJSkVhSTFHeFI3MlhaUVFMTm8zdE5jd3IzVlQxVlpiTgo1cmhHQlVaTFlrMERtd25vWTBCekFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUhuUDJibnJBcWlWCjYzWkpMVzM0UWFDMnRreVFScTNVSUtWR3RVZHFobWRVQ0I1SXRoSUlleUdVRVdqVExpc3BDQzVZRHh4YVdrQjUKTUxTYTlUY0s3SkNOdkdJQUdQSDlILzRaeXRIRW10aFhiR1hJQ3FEVUVmSUVwVy9ObUgvcnBPQUxhYlRvSUVzeQpVNWZPUy9PVVZUM3ZoSldlRjdPblpIOWpnYk1SZG9zVElhaHdQdTEzZEtZMi8zcEtxRW1Cd1JkbXBvTExGbW9MCmVTUFQ4SjREZExGRkh2QWJKalFVbjhKQTZjOHUrMzZJZDIrWE1sTGRZYTdnTnhvZTExQTl6eFJQczRXdlpiMnQKUXZpbHZTbkFWb0ZUSVozSlpjRXVWQXllNFNRY1dKc3FLMlM0UER1VkNFdlg0SmRCRlA2NFhvU08zM3pXaWhtLworMXg3OXZHMUpFcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= service: namespace: kube-system name: edge-health-admission path: /endpoint failurePolicy: Ignore matchPolicy: Exact name: endpoint.k8s.io namespaceSelector: {} objectSelector: {} reinvocationPolicy: Never rules: - apiGroups: - '*' apiVersions: - '*' operations: - UPDATE resources: - endpoints scope: '*' sideEffects: None timeoutSeconds: 5 ``` kube-apiserver會發送AdmissionReview(apiGroup: `admission.k8s.io`，apiVersion：`v1 or v1beta1`)給Webhooks，並封裝成JSON格式，示例如下： ``` # This example shows the data contained in an AdmissionReview object for a request to update the scale subresource of an apps/v1 Deployment { "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview", "request": { # Random uid uniquely identifying this admission call "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", # Fully-qualified group/version/kind of the incoming object "kind": {"group":"autoscaling","version":"v1","kind":"Scale"}, # Fully-qualified group/version/kind of the resource being modified "resource": {"group":"apps","version":"v1","resource":"deployments"}, # subresource, if the request is to a subresource "subResource": "scale", # Fully-qualified group/version/kind of the incoming object in the original request to the API server. # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the # original request to the API server was converted to a version the webhook registered for. "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"}, # Fully-qualified group/version/kind of the resource being modified in the original request to the API server. # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the # original request to the API server was converted to a version the webhook registered for. "requestResource": {"group":"apps","version":"v1","resource":"deployments"}, # subresource, if the request is to a subresource # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the # original request to the API server was converted to a version the webhook registered for. "requestSubResource": "scale", # Name of the resource being modified "name": "my-deployment", # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object) "namespace": "my-namespace", # operation can be CREATE, UPDATE, DELETE, or CONNECT "operation": "UPDATE", "userInfo": { # Username of the authenticated user making the request to the API server "username": "admin", # UID of the authenticated user making the request to the API server "uid": "014fbff9a07c", # Group memberships of the authenticated user making the request to the API server "groups": ["system:authenticated","my-admin-group"], # Arbitrary extra info associated with the user making the request to the API server. # This is populated by the API server authentication layer and should be included # if any SubjectAccessReview checks are performed by the webhook. "extra": { "some-key":["some-value1", "some-value2"] } }, # object is the new object being admitted. # It is null for DELETE operations. "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, # oldObject is the existing object. # It is null for CREATE and CONNECT operations. "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions. # It is null for CONNECT operations. "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...}, # dryRun indicates the API request is running in dry run mode and will not be persisted. # Webhooks with side effects should avoid actuating those side effects when dryRun is true. # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details. "dryRun": false } } ``` 而Webhooks需要向kube-apiserver迴應具有相同版本的AdmissionReview，並封裝成JSON格式，包含如下關鍵欄位： - uid：拷貝傳送給webhooks的AdmissionReview request.uid欄位 - allowed：true表示准許；false表示不准許 - status：當不准許請求時，可以通過status給出相關原因(http code and message) - patch：base64編碼，包含mutating admission webhook對請求物件的一系列[JSON patch操作](https://jsonpatch.com/) - patchType：目前只支援JSONPatch型別 示例如下： ``` # a webhook response to add that label would be： { "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview", "response": {