Kubernetes的負載均衡問題(Nginx Ingress)
阿新 • • 發佈：2017-06-09
targe yaml gen vim bsp 曾經 pla val icc
Kubernetes關於服務的暴露主要是通過NodePort方式，通過綁定minion主機的某個端口,然後進行pod的請求轉發和負載均衡，但這種方式下缺陷是
- Service可能有很多個，如果每個都綁定一個node主機端口的話，主機需要開放外圍一堆的端口進行服務調用，管理混亂
- 無法應用很多公司要求的防火墻規則
理想的方式是通過一個外部的負載均衡器，綁定固定的端口，比如80,然後根據域名或者服務名向後面的Service ip轉發,Nginx很好的解決了這個需求，但問題是如果有新的服務加入，如何去修改Nginx的配置，並且加載這些配置？ Kubernetes給出的方案就是Ingress,Ingress包含了兩大主件Ingress Controller和Ingress.
- Ingress解決的是新的服務加入後，域名和服務的對應問題，基本上是一個ingress的對象，通過yaml進行創建和更新進行加載。
- Ingress Controller是將Ingress這種變化生成一段Nginx的配置，然後將這個配置通過Kubernetes API寫到Nginx的Pod中，然後reload.
具體實現如下:
1.生成一個默認的後端，如果遇到解析不到的URL就轉發到默認後端頁面
[[email protected] ingress]# cat default-backend.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name:default-http-backend labels: k8s-app: default-http-backend namespace: kube-system spec: replicas: 1 template: metadata: labels: k8s-app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any imageis permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: gcr.io/google_containers/defaultbackend:1.0 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: kube-system labels: k8s-app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: k8s-app: default-http-backend
2.部署Ingress Controller
具體文件可以參考官方的
https://github.com/kubernetes/ingress/blob/master/examples/daemonset/nginx/nginx-ingress-daemonset.yaml
這裏貼一個我的
[[email protected] ingress]# cat nginx-ingress-controller.yaml apiVersion: v1 kind: ReplicationController metadata: name: nginx-ingress-lb labels: name: nginx-ingress-lb namespace: kube-system spec: replicas: 1 template: metadata: labels: name: nginx-ingress-lb annotations: prometheus.io/port: ‘10254‘ prometheus.io/scrape: ‘true‘ spec: terminationGracePeriodSeconds: 60 hostNetwork: true containers: - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7 name: nginx-ingress-lb readinessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 timeoutSeconds: 1 ports: - containerPort: 80 hostPort: 80 - containerPort: 443 hostPort: 443 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: KUBERNETES_MASTER value: http://192.168.0.105:8080 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --apiserver-host=http://192.168.0.105:8080
曾經出現的問題是，啟動後pod總是在CrashLoopBack的狀態，通過logs一看發現nginx-ingress-controller的啟動總是去連接apiserver內部集群ip的443端口，導致因為安全問題不讓啟動，後來在args裏面加入
- --apiserver-host=http://192.168.0.105:8080
後成功啟動.
3.配置ingress
配置如下
[[email protected] ingress]# cat dashboard-weblogic.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: dashboard-weblogic-ingress namespace: kube-system spec: rules: - host: helloworld.eric http: paths: - path: /console backend: serviceName: helloworldsvc servicePort: 7001 - path: / backend: serviceName: kubernetes-dashboard servicePort: 80
理解如下:
- host指虛擬出來的域名，具體地址(我理解應該是Ingress-controller那臺Pod所在的主機的地址)應該加入/etc/hosts中,這樣所有去helloworld.eric的請求都會發到nginx
- path:/console匹配後面的應用路徑
- servicePort主要是定義服務的時候的端口，不是NodePort.
- path:/ 匹配後面dashboard應用的路徑,以前通過訪問master節點8080/ui進入dashboard的，但dashboard其實是部署在minion節點中，實際是通過某個路由語句轉發過去而已，dashboard真實路徑如下:
而yaml文件是
[[email protected] ~]# cat kubernetes-dashboard.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: # Keep the name in sync with image version and # gce/coreos/kube-manifests/addons/dashboard counterparts name: kubernetes-dashboard-latest namespace: kube-system spec: replicas: 1 template: metadata: labels: k8s-app: kubernetes-dashboard version: latest kubernetes.io/cluster-service: "true" spec: containers: - name: kubernetes-dashboard image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1 resources: # keep request = limit to keep this container in guaranteed class limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi ports: - containerPort: 9090 args: - --apiserver-host=http://192.168.0.105:8080 livenessProbe: httpGet: path: / port: 9090 initialDelaySeconds: 30 timeoutSeconds: 30 --- kind: Service metadata: name: kubernetes-dashboard namespace: kube-system labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" spec: selector: k8s-app: kubernetes-dashboard ports: - port: 80 targetPort: 9090
所以訪問192.168.51.5:9090端口就會出現dashboard
4.測試
Ok,一切就緒，裝逼開始
訪問http://helloworld.eric/console
訪問http://helloword.eric/ 出現dashboard
5.配置TLS SSL訪問
TLS的配置相當於WebLogic中證書的配置，配置過程如下
- 證書生成
# 生成 CA 自簽證書 mkdir cert && cd cert openssl genrsa -out ca-key.pem 2048 openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca" # 編輯 openssl 配置 cp /etc/pki/tls/openssl.cnf . vim openssl.cnf # 主要修改如下 [req] req_extensions = v3_req # 這行默認註釋關著的 把註釋刪掉 # 下面配置是新增的 [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = helloworld.eric #DNS.2 = kibana.mritd.me # 生成證書 openssl genrsa -out ingress-key.pem 2048 openssl req -new -key ingress-key.pem -out ingress.csr -subj "/CN=helloworld.eric" -config openssl.cnf openssl x509 -req -in ingress.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ingress.pem -days 365 -extensions v3_req -extfile openssl.cnf
需要註意的是DNS需要修改成自己的host名,然後在配置csr證書請求的時候需要將域名或者訪問名帶入subj,比如
-subj "/CN=helloworld.eric"
- 創建secret
kubectl create secret tls ingress-secret --namespace=kube-system --key cert/ingress-key.pem --cert cert/ingress.pem
- 修改Ingress文件啟用證書
[[email protected] ingress]# cat tls-weblogic.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: dashboard-weblogic-ingress namespace: kube-system spec: tls: - hosts: - helloworld.eric secretName: ingress-secret rules: - host: helloworld.eric http: paths: - path: /console backend: serviceName: helloworldsvc servicePort: 7001 - path: / backend: serviceName: kubernetes-dashboard servicePort: 80
- 測試
然後訪問helloworld.eric/console,會自動轉到https頁面，同時查看證書並加入授信列表，可見
訪問helloworld.eric
Kubernetes的負載均衡問題(Nginx Ingress)