[TOC] ## 簡介 Kubernetes 提供了兩種內建的雲端負載均衡機制（ cloud load balancing ）用於釋出公共應用， 工作於傳輸層的 Service 資源，它實現的是 TCP 負載均衡器”，另種是Ingress 資源，它 現的是“ HTTP(S ）負載均衡器” Ingress和Ingress Controller Ingress是Kubernetes API的標準資源型別之一，它其實就是基於DNS名稱(host)或URL路徑把請求轉發至指定的Service資源的規則，用於將叢集外部的請求流量轉發至叢集內部完成服務釋出。然而，**Ingess資源自身並不能執行“流量穿透”，它僅僅是一組路由規則的集合，這些規則要想真正發揮作用還需其它功能的輔助，如監聽某個套接字上，根據這些規則匹配機制路由請求流量：這種能夠為Ingress資源監聽套接字並轉發流量的元件稱之為Ingress控制器** 注意： **不同於 Deployment 控制器等 Ingress 控制器並不直接執行為 kube-controller-rnanager的一部 ，它是Kubemetes叢集的重要附件類似於 CoreDNS 需要在叢集單獨部署** ## 1. 部署Ingress-Controller [官方地址](https://kubernetes.github.io/ingress-nginx/deploy/) 此處部署3.0版本 在你需要部署的node節點上拉去Ingerss-Controller映象 `[root@k8s-master01 daem]# docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0` **進入到GitHub上將mandatory.yaml複製到node節點上** [mandatory.yaml地址](https://github.com/kubernetes/ingress-nginx/tree/nginx-0.30.0/deploy/static) ``` [root@k8s-master01 ingressdeploy]# cat mandatory.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx #data: # whitelist-source-range: 192.168.29.102 #白名單，允許某個IP或IP段的訪問 # block-cidrs: 192.168.29.101 #黑名單拒絕訪問 --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "-" # Here: "-" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: hostNetwork: true ###修改成hostNetwork模式直接共享伺服器的網路名稱空間 # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount nodeSelector: kubernetes.io/os: linux kubernetes.io/hostname: k8s-master02 dnsPolicy: ClusterFirstWithHostNet containers: - name: nginx-ingress-controller imagePullPolicy: IfNotPresent image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 101 runAsUser: 101 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 protocol: TCP #hostPort: 80 - name: https containerPort: 443 protocol: TCP #hostPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown --- apiVersion: v1 kind: LimitRange metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: limits: - min: memory: 90Mi cpu: 100m type: Container ``` 需修改： hostNetwork: true ###修改成hostNetwork模式直接共享伺服器的網路名稱空間 執行create建立Ingress-Controller ```[root@k8s-master01 ingressdeploy]# kubectl create -f mandatory.yaml [root@k8s-master01 ingressdeploy]# kubectl get deploy -n ingress-nginx NAME READY UP-TO-DATE AVAILABLE AGE nginx-ingress-controller 1/1 1 1 76m ``` Ingress-Controller已部署完成 ## 2. 使用Ingress規則 建立測試的web應用 ``` [root@k8s-master01 daem]# cat deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: namespace: default name: nginxapp labels: app: nginx-deploy spec: replicas: 2 selector: matchLabels: app: mynginx template: metadata: labels: app: mynginx spec: containers: - name: nginxweb image: nginx:1.15-alpine [root@k8s-master01 daem]# cat svc.yaml apiVersion: v1 kind: Service metadata: name: nginx-svc labels: app: nginx-svc spec: ports: - name: http port: 80 protocol: TCP nodePort: 30001 #node節點的對映埠 可以通過外部訪問 targetPort: 80 selector: app: mynginx sessionAffinity: None type: NodePort ``` 建立Ingress規則 ``` [root@k8s-master01 daem]# cat ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-daem annotations: kubernetes.io/ingress.class: "nginx" #nginx.ingress.kubernetes.io/limit-connections: 10 #nginx.ingress.kubernetes.io/limit-rate: 100K #nginx.ingress.kubernetes.io/limit-rps: 1 #nginx.ingress.kubernetes.io/limit-rpm: 30 spec: rules: - host: test.nginxsvc.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 path: / ``` 瀏覽器訪問 新增hosts解析 192.168.29.102 test.nginxsvc.com test-tls.test.com  ``` [root@k8s-master01 daem]# curl -I http://test.nginxsvc.com/ HTTP/1.1 200 OK Server: nginx/1.17.8 Date: Thu, 03 Dec 2020 06:56:49 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Vary: Accept-Encoding Last-Modified: Sat, 11 May 2019 00:35:53 GMT ETag: "5cd618e9-264" Accept-Ranges: bytes ``` ### 2.1 Ingress地址重寫 流量重定向到目標URL ``` apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-daem annotations: #kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com #當訪問 test.nginxsvc.com會被重寫到百度上 spec: rules: - host: test.nginxsvc.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 path: / ``` 前後端分離 ``` [root@k8s-master01 daem]# cat ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-daem annotations: #kubernetes.io/ingress.class: "nginx" #nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com nginx.ingress.kubernetes.io/rewrite-target: / #當訪問test.nginxsvc.com/foo 會把請求打到 nginx-svc此service上 spec: rules: - host: test.nginxsvc.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 path: /foo [root@k8s-master01 daem]# cat ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-daem annotations: #kubernetes.io/ingress.class: "nginx" #nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: rules: - host: test.nginxsvc.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 path: /nginxservice(/|$)(.*) paths: - backend: serviceName: tomcat-svc servicePort: 80 path: /tomcatservice(/|$)(.*) #當訪問test.nginxsvc.com:PORT/tomcatservice -> 就會被重定向到tomcat-svc / 資源下 ``` ### 2.2 配置HTTPS ``` [root@k8s-master01 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.cert -subj "/CN=test-tls.test.com/O=test-tls.test.com" Generating a 2048 bit RSA private key ..............................................+++ .........................................................................................+++ writing new private key to 'tls.key' ----- [root@k8s-master01 ~]# kubectl create secret tls ca-cert --key tls.key --cert tls.cert secret/ca-cert created [root@k8s-master01 ~]# cat tlsingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: "false" name: test-tls spec: rules: - host: test-tls.test.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 path: / tls: - hosts: - test-tls.test.com secretName: ca-cert ``` ### 2.3 黑白名單配置 ``` 黑白名單 [root@k8s-master01 ingressdeploy]# cat mandatory.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: #############新增如下資訊################# whitelist-source-range: 192.168.29.102 #白名單，允許某個IP或IP段的訪問 block-cidrs: 192.168.29.101 #黑名單拒絕訪問 ``` ### 2.4 匹配請求頭 ``` apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/server-snippet: | set $agentflag 0; if ($http_user_agent ~* "(Mobile)" ){ set $agentflag 1; } if ( $agentflag = 1 ) { return 301 https://m.example.com; } ``` 解釋： 如果你的http_user_agent == Mobile。那麼就把agentflag set成1 ，然後當agentflag == 1時，就會return 到這個域名 https://m.example.com ### 2.5 速率限制 ``` apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-nginx annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/limit-rate: 100K nginx.ingress.kubernetes.io/limit-rps: 1 nginx.ingress.kubernetes.io/limit-rpm: 30 spec: ...... ``` - nginx.ingress.kubernetes.io/limit-connections 單個IP地址允許的併發連線數。超過此限制時返回503錯誤 - nginx.ingress.kubernetes.io/limit-rps: 每秒從給定IP接受的請求數。突發限制設定為該限制乘以突發乘數，預設乘數為5。當客戶機超過此限制時，將返回limit req status code default:503 - nginx.ingress.kubernetes.io/limit-rpm: 每分鐘從給定IP接受的請求數。突發限制設定為該限制乘以突發乘數，預設乘數為5。當客戶機超過此限制時，將返回limit req status code default:503 - nginx.ingress.kubernetes.io/limit-burst-multiplier: 突發大小限制速率的乘數。預設的突發乘數為5，此批註覆蓋預設乘數。當客戶機超過此限制時，將返回limit req status code default:503。 - nginx.ingress.kubernetes.io/limit-rate-after 初始千位元組數，此後對給定連線的響應的進一步傳輸將受到速率限制。此功能必須在啟用代理緩衝的情況下使用 - nginx.ingress.kubernetes.io/limit-rate 每秒允許傳送到給定連線的KB數。零值禁用速率限制。此功能必須在啟用代理緩衝的情況下使用 - nginx.ingress.kubernetes.io/limit-whitelist 要從速率限制中排除的客戶端IP源範圍。該值是一個逗號分隔的cidr列表 **以上就是Ingress常用的相關配置，所有配置均來自官方文件：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annota