阿里雲Kubernetes實戰1
考慮到公司持續整合與docker容器技術實施已有一段時間,取得了不錯的效果,但對於裝置運維、系統隔離、裝置利用率和擴充套件性還有待提升,綜合目前比較成熟的微服務技術,打算把現有業務遷移到K8S叢集。
由於公司所有業務均部署在阿里雲上,最開始就調研了阿里雲自己提供的Kubernetes叢集,但後來還是放棄了,主要考慮幾方面:
- 阿里雲K8S叢集尚不成熟,使用的版本也相對較老,不能及時更新版本
- 阿里雲K8S叢集目前只支援多主多從結構,同時限定Master節點只能是3個,不能增減,這對於小型業務或者巨型業務均不適用
- 自建原生K8S叢集更有利於拓展和理解整體結構
接下來會詳細介紹在阿里雲搭建原生Kubernetes叢集的過程。
一、K8S叢集搭建
下面的實戰操作基於阿里雲的VPC網路,在4臺ECS上搭建K8S單主多從叢集,部署Gitlab,Gitlab的資料儲存在阿里雲NAS上,服務通過SLB暴露至外網
- 阿里雲VPC * 1
- EIP * 2
- NAT閘道器 * 1
- 共享流量包 * 1
- 阿里雲ECS(無外網IP) * 4
- 阿里雲SLB * 4
- 阿里雲NAS * 1
1.1 VPC組網
對於VPC,新建交換機,目標網段用192.168.0.0/24,4臺ECS的內網IP分別設定為192.168.0.1 ~ 192.168.0.4
1.2 NAT閘道器與EIP打通網路
由於VPC網路內,所有的ECS沒有配置外網IP,所以這裡要配置NAT閘道器和彈性IP來打通外網和VPC的通訊。
- 開通一個NAT閘道器,並加入到VPC內
- 開通兩個EIP,一個用於DNAT(VPC訪問外網),另一個用於SNAT(外網訪問EIP)
- 繫結EIP到NAT閘道器
- 配置DNAT(外網訪問VPC)
- 我們有4臺ECS,每臺機器的22埠分別對映到EIP的不同埠上,如23301~23304,該埠用於SSH訪問ECS
- 同時對映192.168.0.1的6443埠到EIP上,如對映至23443埠,該埠用於訪問K8S叢集的API,見第二章內容
- 配置SNAT(VPC訪問外網)
配置完成後,便可以使用繫結DNAT的EIP的對映埠通過SSH訪問ECS
1.3 使用Kubeasz部署K8S叢集
- 修改hosts檔案,根據實際環境配置master、node、etc的ip
- 這裡將192.168.0.1設定為master,使用單主多從的方式
- 配置完成後重啟所有ECS
二、部署Gitlab實戰
2.1 K8S Dashboard
部署好集群后,我們可以使用DNAT的EIP,通過對映埠23443訪問K8S API和Dashboard
- 進入後會要求輸入API的賬號密碼,與1.3章節hosts檔案裡配置的賬號密碼一致
- 通過賬號密碼驗證後可看到K8S Dashboard登入介面
- 令牌可在Master節點通過以下命令獲取
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
2.2 PV與PVC
K8S中的PV和PVC的概念這裡不再多提,引用官方的一段解釋:
A PersistentVolume
(PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
A PersistentVolumeClaim
(PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., can be mounted once read/write or many times read-only).
在中,我們看到Volumes 有三個,如下表所示
Local location | Container location | Usage |
---|---|---|
/srv/gitlab/data |
/var/opt/gitlab |
For storing application data |
/srv/gitlab/logs |
/var/log/gitlab |
For storing logs |
/srv/gitlab/config |
/etc/gitlab |
For storing the GitLab configuration files |
所以我們也需要給Gitlab for K8S分配3個PV和PVC,這裡我們用到了阿里雲NAS
- 給NAS添加掛載點,選擇VPC網路和VPC的交換機
- 檢視掛載地址
- SSH登入Master節點,掛載NAS,並建立資料夾(注意PV的path必須已存在才可以成功建立,所以需要先在NAS中建立資料夾)
mkdir /nas sudo mount -t nfs -o vers=4.0 xxx.xxx.nas.aliyuncs.com:/ /nas mkdir -p /gitlab/data mkdir -p /gitlab/logs mkdir -p /gitlab/config
- 編寫PV和PVC的YAML,根據實際需求替換server節點的NAS掛載地址配置以及storage大小配置
apiVersion v1 kind Namespace metadata name gitlab labels name gitlab --- apiVersion v1 kind PersistentVolume metadata name gitlab-data labels release gitlab-data namespace gitlab spec capacity storage 500Gi accessModes ReadWriteMany persistentVolumeReclaimPolicy Retain nfs path /gitlab/data server xxx.xxx.nas.aliyuncs.com --- apiVersion v1 kind PersistentVolume metadata name gitlab-config labels release gitlab-config namespace gitlab spec capacity storage 1Gi accessModes ReadWriteMany persistentVolumeReclaimPolicy Retain nfs path /gitlab/config server xxx.xxx.nas.aliyuncs.com --- apiVersion v1 kind PersistentVolume metadata name gitlab-log labels release gitlab-log namespace gitlab spec capacity storage 1Gi accessModes ReadWriteMany persistentVolumeReclaimPolicy Retain nfs path /gitlab/log server xxx.xxx.nas.aliyuncs.com --- apiVersion v1 kind PersistentVolumeClaim metadata name gitlab-data-claim namespace gitlab spec accessModes ReadWriteMany resources requests storage 500Gi selector matchLabels release gitlab-data --- apiVersion v1 kind PersistentVolumeClaim metadata name gitlab-config-claim namespace gitlab spec accessModes ReadWriteMany resources requests storage 1Gi selector matchLabels release gitlab-config --- apiVersion v1 kind PersistentVolumeClaim metadata name gitlab-log-claim namespace gitlab spec accessModes ReadWriteMany resources requests storage 1Gi selector matchLabels release gitlab-log
2.3 K8S部署Gitlab
接下來補全Gitlab的Deployment和Service
apiVersion apps/v1 kind Deployment metadata name gitlab namespace gitlab spec selector matchLabels app gitlab replicas1 strategy type Recreate template metadata labels app gitlab spec containers image gitlab/gitlab-ce latest name gitlab ports containerPort80 name gitlab-http containerPort443 name gitlab-https containerPort22 name gitlab-ssh volumeMounts name gitlab-config mountPath /etc/gitlab name gitlab-log mountPath /var/log/gitlab name gitlab-data mountPath /var/opt/gitlab volumes name gitlab-data persistentVolumeClaim claimName gitlab-data-claim name gitlab-config persistentVolumeClaim claimName gitlab-config-claim name gitlab-log persistentVolumeClaim claimName gitlab-log-claim --- kind Service apiVersion v1 metadata name gitlab-service labels app gitlab-service namespace gitlab spec selector app gitlab ports protocol TCP name gitlab-https port443 targetPort443 protocol TCP name gitlab-http port80 targetPort80 --- kind Service apiVersion v1 metadata name gitlab-ssh-service labels app gitlab-ssh-service namespace gitlab spec type NodePort selector app gitlab ports protocol TCP name gitlab-ssh port22 targetPort22 nodePort30000
- 注意在Deployment中,開放了Gitlab Pod的80、443和22埠,用於Gitlab的HTTP、HTTPS和SSH的訪問
- 建立了2個Service,第一個只將80和443埠開放到Cluster IP上,第二個Service通過NodePort將22埠對映到NodeIp的30000埠上
- 我們將2.2章節PV與PVC中的相關程式碼和上面的程式碼合併,並命名成gitlab.yaml,上傳到Master節點,執行命令
kubectl apply -f gitlab.yaml
- 接下來進入Gitlab的Pod,修改gitlab的域名,並啟用https訪問
kubectl get pod --namespace=gitlab # 獲得gitlab pod名稱後 kubectl exec -it gitlab-xxxx-xxxx --namespace=gitlab /bin/bash # 進入pod後 vi /etc/gitlab/gitlab.rb # 修改external_url 'https://xxx.xxx.com',儲存後退出 gitlab-ctl reconfigure exit
到這裡,配置與部署基本完成了,但我們還不能從外網訪問Gitlab,不過至少可以在叢集內驗證配置是否正確。
- 在Master節點檢視Service
kubectl get svc --namespace=gitlab
可以看到443和80埠已經開發給Cluster IP,同時22埠對映到了30000的NodePort上
- 通過curl命令檢視訪問結果
curl https://10.68.88.97 --insecure
這時返回一串包含redirect的字元,如下
<html><body>You are being <a href="https://10.68.88.97/users/sign_in">redirected</a>.</body></html>
表示服務已部署成功
- 如果有telnet客戶端,還可以驗證30000埠,在任何一個節點上執行任意一條命令
telnet 192.168.0.1:30000 telnet 192.168.0.2:30000 telnet 192.168.0.3:30000 telnet 192.168.0.4:30000
2.4 使用Ingress-Nginx和阿里雲SLB暴露服務
K8S暴露服務的方法有3種:
- ClusterIP:叢集內可訪問,但外部不可訪問
- NodePort:通過NodeIP:NodePort方式可以在叢集內訪問,結合EIP或者雲服務VPC負載均衡也可在叢集外訪問,但開放NodePort一方面不安全,另一方面隨著應用的增多不方便管理
- LoadBalancer:某些雲服務提供商會直接提供LoadBalancer模式,將服務對接到負載均衡,其原理是基於kubernetes的controller做二次開發,並整合到K8S叢集,使得叢集可以與雲服務SDK互動
由於我們的叢集搭建在阿里雲上,所以第一時間想到的是LoadBalancer方案,但很遺憾,沒辦法使用,原因如下:
迴歸到NodePort的方式,目前已有的解決方案是基於Ingress的幾款工具,如Ingress-Nginx、Traefik-Ingress,他們的對比如下(注意,目前的版本是IngressNginx 0.13.0、Traefik 1.6):
- IngressNginx和Traefik都是通過hostname方式反向代理已解決埠暴露問題
- IngressNginx依賴於Nginx,功能更多;Traefik不依賴Nginx,所以更輕量
- IngressNginx支援4層和7層LB,但4層也不好用,Traefik只支援7層代理
- 目前網上關於IngressNginx的文章都是beta 0.9.X版本的資訊,而IngressNginx在Github的地址也變化了,直接由Kubernetes維護,所以網上的文章基本沒參考性,還需看官方文件,但是官方文件極其混亂和不完善!!! 後面會有填坑指南。
最終我們還是選擇了Ingress-Nginx,結合阿里雲SLB,最終的拓撲圖如下所示:
其原理是:
- 通過Service的ClusterIP負載Pod
- 通過Ingress-Nginx監聽Ingress配置,動態生成Nginx,並將Nginx暴露到23456的NodePort
- 通過阿里雲SLB監聽所有節點的23456埠
接下來看詳細步驟。
2.4.1 部署Ingress-Nginx
- 替換gcr.io的映象為阿里雲映象
- 暴露服務埠到NodePort 23456
- 整合成一個ingress-nginx.yaml
apiVersion v1 kind Namespace metadata name ingress-nginx --- apiVersion extensions/v1beta1 kind Deployment metadata name default-http-backend labels app default-http-backend namespace ingress-nginx spec replicas1 selector matchLabels app default-http-backend template metadata labels app default-http-backend spec terminationGracePeriodSeconds60 containers name default-http-backend # Any image is permissible as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image registry.cn-shenzhen.aliyuncs.com/heygears/defaultbackend1.4 livenessProbe httpGet path /healthz port8080 scheme HTTP initialDelaySeconds30 timeoutSeconds5 ports containerPort8080 resources limits cpu 10m memory 20Mi requests cpu 10m memory 20Mi --- apiVersion v1 kind Service metadata name default-http-backend namespace ingress-nginx labels app default-http-backend spec ports port80 targetPort8080 selector app default-http-backend --- kind ConfigMap apiVersion v1 metadata name nginx-configuration namespace ingress-nginx labels app ingress-nginx --- kind ConfigMap apiVersion v1 metadata name tcp-services namespace ingress-nginx --- kind ConfigMap apiVersion v1 metadata name udp-services namespace ingress-nginx --- apiVersion v1 kind ServiceAccount metadata name nginx-ingress-serviceaccount namespace ingress-nginx --- apiVersion rbac.authorization.k8s.io/v1beta1 kind ClusterRole metadata name nginx-ingress-clusterrole rules apiGroups "" resources configmaps endpoints nodes pods secrets verbs list watchapiGroups "" resources nodes verbs getapiGroups "" resources services verbs get list watchapiGroups "extensions" resources ingresses verbs get list watchapiGroups "" resources events verbs create patchapiGroups "extensions" resources ingresses/status verbs update--- apiVersion rbac.authorization.k8s.io/v1beta1 kind Role metadata name nginx-ingress-role namespace ingress-nginx rules apiGroups "" resources configmaps pods secrets namespaces verbs getapiGroups "" resources configmaps resourceNames # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. "ingress-controller-leader-nginx" verbs get updateapiGroups "" resources configmaps verbs createapiGroups "" resources endpoints verbs get --- apiVersion rbac.authorization.k8s.io/v1beta1 kind RoleBinding metadata name nginx-ingress-role-nisa-binding namespace ingress-nginx roleRef apiGroup rbac.authorization.k8s.io kind Role name nginx-ingress-role subjects kind ServiceAccount name nginx-ingress-serviceaccount namespace ingress-nginx --- apiVersion rbac.authorization.k8s.io/v1beta1 kind ClusterRoleBinding metadata name nginx-ingress-clusterrole-nisa-binding roleRef apiGroup rbac.authorization.k8s.io kind ClusterRole name nginx-ingress-clusterrole subjects kind ServiceAccount name nginx-ingress-serviceaccount namespace ingress-nginx --- apiVersion extensions/v1beta1 kind Deployment metadata name nginx-ingress-controller namespace ingress-nginx spec replicas1 selector matchLabels app ingress-nginx template metadata labels app ingress-nginx annotations prometheus.io/port'10254' prometheus.io/scrape'true' spec serviceAccountName nginx-ingress-serviceaccount containers name nginx-ingress-controller image registry.cn-shenzhen.aliyuncs.com/heygears/nginx-ingress-controller0.13.0