8.5 Ingress實現基於域名的多虛擬主機、URL轉發、及多域名https實現等案例
阿新 • • 發佈：2021-11-29
1.什麼是Ingress
Ingress 公開了從k8s叢集外部到叢集內服務的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 資源上定義的規則控制。
可以將 Ingress 配置為服務提供外部可訪問的 URL、負載均衡流量、終止 SSL/TLS，以及提供基於名稱的虛擬主機等能力
ingress具體的工作原理如下:
step1：ingress contronler通過與k8s的api進行互動，動態的去感知k8s叢集中ingress服務規則的變化，然後讀取它，並按照定義的ingress規則，轉發到k8s叢集中對應的service。
step2：而這個ingress規則寫明瞭哪個域名對應k8s叢集中的哪個service，然後再根據ingress-controller中的nginx配置模板，生成一段對應的nginx配置。
step3：然後再把該配置動態的寫到ingress-controller的pod裡，該ingress-controller的pod裡面執行著一個nginx服務，控制器會把生成的nginx配置寫入到nginx的配置檔案中，然後reload一下，使其配置生效,以此來達到域名分配置及動態更新的效果。
ingress可以解決的問題
1）動態配置服務
如果按照傳統方式, 當新增加一個服務時, 我們可能需要在流量入口加一個反向代理指向我們新的k8s服務. 而如果用了Ingress, 只需要配置好這個服務, 當服務啟動時, 會自動註冊到Ingress的中, 不需要而外的操作。
2）減少不必要的埠暴露
配置過k8s的都清楚, 第一步是要關閉防火牆的, 主要原因是k8s的很多服務會以NodePort方式映射出去, 這樣就相當於給宿主機打了很多孔, 既不安全也不優雅. 而Ingress可以避免這個問題, 除了Ingress自身服務可能需要映射出去, 其他服務都不要用NodePort方式。
原文連結：https://blog.csdn.net/weixin_44729138/article/details/105978555
2.ingress控制器部署
為了讓 Ingress 資源工作，叢集必須有一個正在執行的 Ingress 控制器
與作為 kube-controller-manager 可執行檔案的一部分執行的其他型別的控制器不同， Ingress 控制器不是隨叢集自動啟動的。 基於此頁面，你可選擇最適合你的叢集的 ingress 控制器實現。
Kubernetes 作為一個專案，目前支援和維護 AWS， GCE 和 nginx Ingress 控制器。
但是基於nginx服務的ingress controller根據不同的開發公司，又分為k8s社群的ingres-nginx和nginx公司的nginx-ingress。
在此根據github上的活躍度和關注人數，我們選擇的是k8s社群的ingres-nginx進行部署測試
2.1 準備ingress控制器部署使用的docker映象(v1.0.0的版本)
docker pull willdockerhub/ingress-nginx-controller:v1.0.0
docker tag 192.168.1.110/base/ingress-nginx-controller:v1.0.0
docker push 192.168.1.110/base/ingress-nginx-controller:v1.0.0
docker pull jettech/kube-webhook-certgen:v1.0.0
docker tag 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
docker push 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
2.2 修改ingress-controller-deploy.yaml
1.官方檔案中建立的controller service 的type是LoadBalancer，改成NodePort，並新增相應的埠
2.將檔案中的映象替換為本地harbor中的映象
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
#type: LoadBalancer
type: NodePort
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 40080
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 40443
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
點選檢視完整的deploy.yaml檔案
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
#type: LoadBalancer
type: NodePort
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 40080
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 40443
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
containers:
- name: controller
image: 192.168.1.110/base/ingress-nginx-controller:v1.0.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
2.3 建立ingress-controller
kubectl apply -f ingress-controller-deploy-v1.0.0.yml
2.4 檢視相關資源是否建立成功
root@k8-deploy:~/k8s-yaml/ingress# kubectl get svc,pod -n ingress-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/ingress-nginx-controller NodePort 10.0.2.50 <none> 80:40080/TCP,443:40443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission ClusterIP 10.0.135.64 <none> 443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/ingress-nginx-admission-create-mr6qt 0/1 Completed 0 21h 10.100.112.44 192.168.2.19 <none> <none>
pod/ingress-nginx-admission-patch-sxrl6 0/1 Completed 1 21h 10.100.112.45 192.168.2.19 <none> <none>
pod/ingress-nginx-controller-556976fb4d-dljbp 1/1 Running 0 21h 10.100.224.73 192.168.2.18 <none> <none>
# 檢視ingress-nginx-controller 版本
# kubectl exec -it ingress-nginx-controller-556976fb4d-dljbp -n ingress-nginx -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.0.0
Build: 041eb167c7bfccb1d1653f194924b0c5fd885e10
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.20.1
-------------------------------------------------------------------------------
# 檢視node節點上是否監聽ingress-controller的40080,40443 NodePort埠
root@k8-deploy:~/k8s-yaml/ingress# for i in 17 18 19;do echo node:192.168.1.$i;ssh 192.168.2.$i -C "netstat -ntlp |grep -E '40080|40443'";echo;done
node:192.168.1.17
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 3452053/kube-proxy
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 3452053/kube-proxy
node:192.168.1.18
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 1364498/kube-proxy
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 1364498/kube-proxy
node:192.168.1.19
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 251137/kube-proxy
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 251137/kube-proxy
2.5 配置haproxy（邊緣節點） 代理ingress
# cat /etc/haproxy/haproxy.cfg
...
frontend ingress
bind 192.168.2.110:80
mode tcp
log global
default_backend ingress-servers
frontend ingress-https
bind 192.168.2.110:443
mode tcp
log global
default_backend ingress-servers-https
backend ingress-servers
balance source
server websrv1 192.168.2.17:40080 check maxconn 2000
server websrv2 192.168.2.18:40080 check maxconn 2000
server websrv3 192.168.2.19:40080 check maxconn 2000
backend ingress-servers-https
balance source
server websrv1 192.168.2.17:40443 check maxconn 2000
server websrv2 192.168.2.18:40443 check maxconn 2000
server websrv3 192.168.2.19:40443 check maxconn 2000
3 基於域名的多虛擬主機的測試
下圖顯示了客戶端是如果通過 Ingress Controller 連線到其中一個 Pod 的流程，客戶端首先對 域名 執行 DNS 解析，得到 Ingress Controller 所在節點的 IP，然後客戶端向 Ingress Controller 傳送 HTTP 請求，然後根據 Ingress 物件裡面的描述匹配域名，找到對應的 Service 物件，並獲取關聯的 Endpoints 列表，將客戶端的請求轉發給其中一個 Pod。
3.1 準備2個域名和映象
為方便測試，域名放到hosts檔案中
192.168.2.110 app1.test.com app2.test.com
測試的預期效果為訪問不通的域名，會返回不同的內容。
2個映象製作如下:
基礎映象使用之前已經做好的nginx映象(應程式碼釋出目錄改為了/usr/share/nginx/html/)
nginx-app1# ll
-rw-r--r-- 1 root root 83 11月 11 13:54 Dockerfile
-rw-r--r-- 1 root root 23 11月 11 13:54 index.html
# cat Dockerfile
FROM 192.168.1.110/web/nginx-set-config:v1
COPY index.html /usr/share/nginx/html/
# cat index.html
nginx test html : app1
# docker build -t 192.168.1.110/web/nginx-set-config-app1:v1 .
# docker push 192.168.1.110/web/nginx-set-config-app1:v1
## ==========================
nginx-app2# ll
-rw-r--r-- 1 root root 83 11月 11 13:56 Dockerfile
-rw-r--r-- 1 root root 23 11月 11 13:56 index.html
# cat Dockerfile
FROM 192.168.1.110/web/nginx-set-config:v1
COPY index.html /usr/share/nginx/html/
# cat index.html
nginx test html : app2
# docker build -t 192.168.1.110/web/nginx-set-config-app2:v1 .
# docker push 192.168.1.110/web/nginx-set-config-app2:v1
3.2 編寫2個service的yaml檔案
root@k8-deploy:~/k8s-yaml/ingress# cat nginx-app1-svc.yml
apiVersion: v1
kind: Service
metadata:
name: nginx-app1-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 40003
protocol: TCP
selector:
app: nginx-app1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-app1-deployment
namespace: yan-test
labels:
app: nginx-app1
spec:
replicas: 1
selector:
matchLabels:
app: nginx-app1
template:
metadata:
labels:
app: nginx-app1
spec:
containers:
- name: nginx-app1-ct
image: 192.168.1.110/web/nginx-set-config-app1:v1
ports:
- containerPort: 80
## =========================
root@k8-deploy:~/k8s-yaml/ingress# cat nginx-app2-svc.yml
apiVersion: v1
kind: Service
metadata:
name: nginx-app2-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 40004
protocol: TCP
selector:
app: nginx-app2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-app2-deployment
namespace: yan-test
labels:
app: nginx-app2
spec:
replicas: 1
selector:
matchLabels:
app: nginx-app2
template:
metadata:
labels:
app: nginx-app2
spec:
containers:
- name: nginx-app2-ct
image: 192.168.1.110/web/nginx-set-config-app2:v1
ports:
- containerPort: 80
3.3 建立service
# kubectl apply -f nginx-app1-svc.yml
# kubectl apply -f nginx-app2-svc.yml
# kubectl get svc -n yan-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-app1-svc NodePort 10.0.7.177 <none> 80:40003/TCP 23h
nginx-app2-svc NodePort 10.0.63.21 <none> 80:40004/TCP 23h
3.4 編寫2個域名對應的2個service的ingress配置檔案
# cat ingress-muti-host-nginx.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-web
namespace: yan-test
annotations:
kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的型別
nginx.ingress.kubernetes.io/use-regex: "true" ##指定後面rules定義的path可以使用正則表示式
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##連線超時時間,預設為5s
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##後端伺服器迴轉資料超時時間,預設為60s
nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##後端伺服器響應超時時間,預設為60s
nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客戶端上傳檔案，最大大小，預設為20m
#nginx.ingress.kubernetes.io/rewrite-target: / ##URL重寫
#nginx.ingress.kubernetes.io/app-root: /index.html
spec:
rules:
- host: app1.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app1-svc
port:
number: 80
- host: app2.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app2-svc
port:
number: 80
3.5 建立ingress規則
# kubectl apply -f ingress-muti-host-nginx.yml
# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-web <none> app1.test.com,app2.test.com 10.0.2.50 80 21h
3.6 使用域名訪問測試
root@k8-deploy:~# curl app1.test.com
nginx test html : app1
root@k8-deploy:~# curl app2.test.com
nginx test html : app2
4 單域名多URL轉發
預期效果為訪問tomcat.test.com 返回tomato預設頁面內容，訪問tomcat.test.com/app1/index.html返回app1專案的內容
4.1 構建tomcat映象
tomcat-app1# ll
總用量 20
drwxr-xr-x 3 root root 4096 11月 12 13:42 ./
drwxr-xr-x 6 root root 4096 11月 11 13:56 ../
drwxr-xr-x 2 root root 4096 11月 10 16:24 app1/
-rw-r--r-- 1 root root 164 11月 10 16:24 app1.tgz
-rw-r--r-- 1 root root 96 11月 10 16:23 Dockerfile
root@k8-deploy:~/k8s-yaml/ingress/images/tomcat-app1# cat Dockerfile
FROM 192.168.1.110/web/alpine-jdk-8u192-tomcat-8.5.70:v20211020-1014
ADD app1.tgz /opt/webapps
root@k8-deploy:~/k8s-yaml/ingress/images/tomcat-app1# cat app1/index.html
tomcat app1 html
# docker build -t 192.168.1.110/web/tomcat-app1:v1 .
# docker push 192.168.1.110/web/tomcat-app1:v1
4.2 編寫svc ymal並建立tomcat svc
# cat tomcat-app1-svc.yml
apiVersion: v1
kind: Service
metadata:
name: tomcat-app1-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 40001
protocol: TCP
selector:
app: tomcat-app1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-app1-deployment
namespace: yan-test
labels:
app: tomcat-app1
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-app1
template:
metadata:
labels:
app: tomcat-app1
spec:
containers:
- name: tomcat-app1-ct
image: 192.168.1.110/web/tomcat-app1:v1
ports:
- containerPort: 8080
4.3 建立tomcat svc並檢視
# kubectl apply -f tomcat-app1-svc.yml
# kubectl get svc,pod -n yan-test -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/tomcat-app1-svc NodePort 10.0.144.64 <none> 80:40001/TCP 24h app=tomcat-app1
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/tomcat-app1-deployment-754855ff99-jfqwz 1/1 Running 0 24h 10.100.224.72 192.168.2.18 <none> <none>
4.4 編寫ingress yaml
# cat ingress-muti-path-tomcat.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tomcat-ingress
namespace: yan-test
annotations:
kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的型別
nginx.ingress.kubernetes.io/use-regex: "true" ##指定後面rules定義的path可以使用正則表示式
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##連線超時時間,預設為5s
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##後端伺服器迴轉資料超時時間,預設為60s
nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##後端伺服器響應超時時間,預設為60s
nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客戶端上傳檔案，最大大小，預設為20m
#nginx.ingress.kubernetes.io/rewrite-target: / ##URL重寫
#nginx.ingress.kubernetes.io/app-root: /index.html
spec:
rules:
- host: tomcat.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tomcat-app1-svc
port:
number: 80
- path: /app1
pathType: Prefix
backend:
service:
name: tomcat-app1-svc
port:
number: 80
4.5 建立tomcat ingress規則
# kubectl apply -f ingress-muti-path-tomcat.yml
# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
tomcat-ingress <none> tomcat.test.com 10.0.2.50 80 26m
4.6 通過域名和不同的url進行驗證
# curl tomcat.test.com
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Apache Tomcat/8.5.70</title>
<link href="favicon.ico" rel="icon" type="image/x-icon" />
<link href="tomcat.css" rel="stylesheet" type="text/css" />
</head>
...
# curl tomcat.test.com/app1/index.html
tomcat app1 html
5 ingress https配置測試
5.1 準備ssl證書
申請證書的域名需要完成ICP備案，使用阿里雲免費的ssl了證書
root@k8-deploy:~/k8s-yaml/ingress/cert# ll
-rw-r--r-- 1 root root 1679 11月 16 15:02 6611016_ingress.t.faxuan.net.key
-rw-r--r-- 1 root root 4153 11月 16 15:02 6611016_ingress.t.faxuan.net_nginx.zip
-rw-r--r-- 1 root root 3813 11月 16 15:02 6611016_ingress.t.faxuan.net.pem
5.2 將ssl證書建立為secret引入k8s
root@k8-deploy:~/k8s-yaml/ingress/cert# kubectl create secret tls ingress.t.faxuan.net-secret \
> --key=./6611016_ingress.t.faxuan.net.key \
> --cert=./6611016_ingress.t.faxuan.net.pem \
> -n yan-test
secret/ingress.t.faxuan.net-secret created
root@k8-deploy:~/k8s-yaml/ingress/cert# kubectl get secrets -n yan-test
NAME TYPE DATA AGE
default-token-8fd2j kubernetes.io/service-account-token 3 68m
ingress.t.faxuan.net-secret kubernetes.io/tls 2 5s
5.3 編寫 ingress htts規則的yaml
後端的server 使用之前已經建立的nginx-app1-svc
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
namespace: yan-test
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
spec:
ingressClassName: nginx
tls:
- hosts:
- ingress.t.faxuan.net
secretName: ingress.t.faxuan.net-secret
rules:
- host: ingress.t.faxuan.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app1-svc
port:
number: 80
5.4 使用yaml檔案建立ingress規則
kubectl apply -f ingress-https-single.yml
root@k8-deploy:~/k8s-yaml/ingress# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp nginx ingress.t.faxuan.net 10.0.233.49 80, 443 17m
遇到的錯誤：
Error from server (InternalError): error when creating "ingress-https-single.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": the server rejected our request for an unknown reason
2種解決方法：
1.使用ingress的v1beta1版本格式的yaml檔案
2.刪除 ValidatingWebhookConfiguration資源中ingress-nginx-admission
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
參考：https://knowledge.broadcom.com/external/article/227955/installation-reported-during-nginx-setup.html
5.5 haproxy 代理配置
listen ingress-443
bind *:443
mode tcp
server k8s1 192.168.2.17:40443 check inter 3s fall 3 rise 5
server k8s2 192.168.2.18:40443 check inter 3s fall 3 rise 5
server k8s3 192.168.2.19:40443 check inter 3s fall 3 rise 5
# 重啟haproxy
systemctl restart haproxy
5.6 驗證ingress https
域名解析，比如：ingress.t.faxuan.net解析到haproxy所在的伺服器IP