# 【k8s實戰一】Jenkins 部署應用到 Kubernetes ## 01 本文主旨 目標是演示整個Jenkins從原始碼構建映象到部署映象到Kubernetes叢集過程。 為了簡化流程與容易重現文中效果，做出如下操作： - 使用 `VBox` 建立兩臺虛擬機器 - 使用 `Minikube` 初始化簡單的k8s叢集 - 使用 `GitHub` 作為程式碼倉庫，本文內容所有配置檔案均會上傳至此倉庫 https://github.com/hellxz/cicd-demo.git ## 02 CI/CD流程  根據我畫的這張圖，Jenkins任務整體的流程分為五個步驟： - 獲取原始碼 - 構建制品 - 構建映象並推送 - 發起部署請求 - k8s排程pod，拉取映象，部署完成 ## 03 準備虛擬機器 準備兩個虛擬機器，均安裝好docker，參考[Ubuntu 16.04及以上 安裝/解除安裝 Docker-CE](https://www.cnblogs.com/hellxz/p/10417549.html) | VM用途 | 系統版本 | IP | | ---------- | ------------------ | -------------- | | Jenkins | Ubuntu 18.04.2 LTS | 192.168.87.128 | | k8s master | Ubuntu 18.04.2 LTS | 192.168.87.129 | 設定Jenkins機器免密登入k8s master主機 ```bash ssh-keygen -t rsa ssh-copy-id [email protected] ```  驗證免密效果 ```bash ssh [email protected] exit ``` ## 04 部署registry 部署registry之前，可以先將私有倉庫配置到 /etc/docker/daemon.json，兩臺虛擬機器均需要配置 ```bash sudo mkdir -p /etc/docker sudo vim /etc/docker/daemon.json ``` 內容如下： ```javascript { "insecure-registries": ["192.168.87.129:5000"] } ``` 接著在 192.168.87.129 虛擬機器上部署registry，*生產環境推薦使用Harbor* ```bash docker run -d --name registry -p 5000:5000 registry ```  ## 05 配置maven 在jenkins虛擬機器安裝maven ```bash wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz tar zxvf apache-maven-3.6.3-bin.tar.gz mv apache-maven-3.6.3 maven-3.6.3 ``` 修改 `/etc/profile`，新增 `M2_HOME` 環境變數對映到 maven 目錄，重寫 `PATH` 新增 maven 的 bin 目錄，如圖31、32行  ## 06 Tomcat部署Jenkins 在jenkins虛擬機器 192.168.87.128上執行命令下載 tomcat 與 jenkins.war ```bash #下載tomcat wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz #解壓並重命名 tar -zxf apache-tomcat-9.0.41.tar.gz mv apache-tomcat-9.0.41 /home/hellxz/jenkins #進入jenkins安裝目錄 cd ~/jenkins/webapps #刪除webapps下示例專案 rm -rf * #下載最新版jenkins.war wget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war #安裝openjdk、git sudo apt-update sudo apt-get install -y default-jdk default-jdk-headless git ``` 設定Jenkins家目錄，編輯 /etc/profile，新增 `JENKINS_HOME` 環境變數  重新整理配置 ```bash source /etc/profile ``` 修改tomcat下 `conf/context.xml`，加大靜態資源快取，提前解決啟動 Jenkins 時的報錯 ~~~xml ~~~  啟動Jenkins ```bash cd ~/jenkins/bin ./startup.sh ```  訪問 http://192.168.87.129:8080，安裝推薦外掛，安裝加速可參考 [Jenkins安裝外掛提速](https://www.cnblogs.com/hellxz/p/jenkins_install_plugins_faster.html) ## 07 部署Kubernetes叢集 部署一個可以通過 `kubectl` 命令部署應用的k8s環境即可，部署k8s環境不是本文重點，以下為參考

使用kubeadm部署Kubernetes叢集，點選展開檢視

參考我之前的部落格吧，差異應該不大
【k8s學習筆記】使用 kubeadm 部署 v1.18.5 版本 Kubernetes叢集

使用Minikube部署單節點測試k8s環境，點選展開檢視

#新增阿里apt kubernetes源
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
#安裝kubectl
sudo apt-get install -y kubectl
#下載minikube二進位制檔案
curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
#部署k8s環境，預設使用docker驅動。讀者也可使用vbox
minikube start -–insecure-registry=192.168.87.129:5000
 

> 本文使用Minikube做測試，坑比較多，想趟坑可以先參考下 https://www.kubernetes.org.cn/6388.html > >

>

關於Minikube的吐槽

>

> 寫文時，用的Win10建立Linux無桌面虛擬機器，在虛擬機器基礎上裝的minikube，minikube預設使用Docker在虛擬機器中又建立了個Docker容器，裡邊才是k8s環境，所以想在虛擬機器外訪問套了兩層的內網應用服務，可能還需要在虛擬機器中加Nginx，通過它轉發請求到虛擬機器中的內網地址……雖然瘋狂套娃挺坑的，但為了演示下效果還是可以的。 > PS：在Linux/MacOS環境直接裝Minikube，是可以通過瀏覽器直接看的。由於未直接在Win環境使用Docker Desktop，所以未確定Win下是否可用。

>

## 08 準備Demo Demo分五部分：測試程式碼、Jenkins構建指令碼、docker映象構建指令碼、k8s部署配置yaml、部署指令碼，均處在同一倉庫中 倉庫地址：https://github.com/hellxz/cicd-demo.git 目錄結構：  ### demo 使用start.spring.io生成的只新增spring-boot-starter-web依賴，在啟動類上加了個 `/test` 的介面  ### Dockerfile ```bash FROM openjdk:8-jdk #設定工作目錄 WORKDIR /root #複製製品jar到/root/app.jar位置 ADD target/*.jar app.jar #JVM引數，給後期調優使用 ENV JVM_OPTS="" #啟動服務 CMD java ${JVM_OPTS} \ -Djava.security.egd=file:/dev/./urandom \ -jar app.jar ``` ### shell `artifact2image.sh` 製品轉映象Shell指令碼，本文為了簡單未安裝Jenkins關於docker的外掛 ```bash #!/bin/bash # -- 構建映象並推送私有倉庫 -- set -eu #如有報錯或取不到變數情況停止執行 #聲名常量 IMG_REGISTRY="192.168.87.129:5000" #映象倉庫 IMG_NAME="cicd-demo" IMG_TAG=`date "+%Y%M%d_%H%M"` #映象標籤，如 20201223_1351 IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #映象上傳與拉取的名稱 #構建映象 docker build -t ${IMG_FULL_NAME} . #推送映象 docker push ${IMG_FULL_NAME} #刪除本地映象 docker rmi ${IMG_FULL_NAME} #修改deploy.yaml的映象標籤 sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml ``` `deploy2k8s.sh` 部署映象到k8s環境指令碼 ```bash #!/bin/bash # -- 部署image到k8s -- # !注意：需要提前做ssh免密登入 set -eu #定義常量 PROJECT_NAME="cicd-demo" UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}" FILE_NAME="${UPLOAD_DIR}/deploy.yaml" SSH_USER="hellxz" SSH_IP="192.168.87.129" #首先刪除待上傳目錄的同名檔案 ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}" #確保部署檔案目錄存在 ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}" #遠端複製部署檔案 scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME} #遠端執行部署命令 ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}" ``` ### Jenkinsfile ```groovy pipeline{ agent any stages { //由於原始碼和Jenkinsfile處於同一倉庫，在Jenkins專案執行時，會先將Jenkinsfile所在的倉庫克隆下來，為了簡單，這裡就不重複新增拉取原始碼的操作了 //其它情況，如使用多個倉庫一些構建的，這種就需要額外新增拉取程式碼的stage了。 stage('Build Artifact') { steps{ sh label:'maven building', script: 'mvn clean package -DskipTests' } } stage('Build Image'){ steps{ sh label:'image building', script: '/bin/bash artifact2image.sh' } } stage('Deploy k8s'){ steps{ sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh' } } } post { success{ //成功清理工作空間，失敗保留現場 cleanWs() } } } ``` ### deploy.yaml ```yaml apiVersion: apps/v1 #api版本 kind: Deployment #物件型別 metadata: #元資訊標識 name: cicd-demo namespace: default labels: #自定義Deployment繫結的標籤 app: cicd-demo spec: #期待執行狀態 replicas: 1 #部署例項數 selector: matchLabels: app: cicd-demo template: #pod metadata: labels: #與spec.selector.matchLabels相同 app: cicd-demo spec: #pod容器期望的狀態 containers: - name: cicd-demo image: {{IMAGE_NAME}} imagePullPolicy: IfNotPresent env: - name: JVM_OPTS value: "-Xms128m -Xmx256m" ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: cicd-demo-svc namespace: default labels: service: cicd-demo-svc spec: selector: #匹配pod標籤 app: cicd-demo type: NodePort #svc型別，ExternalName, ClusterIP, NodePort, and LoadBalancer ports: - name: cicd-demo-port protocol: TCP port: 8080 nodePort: 30000 ``` ## 09 建立Jenkins流水線專案 訪問 http://192.168.87.128:8080/jenkins 並登入，以下建立新流水線demo構建專案  ## 部署與驗證 執行構建demo程式   日誌輸出： ```bash Started by user admin Obtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.git Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE No credentials specified Cloning the remote Git repository Cloning repository https://github.com/hellxz/cicd-demo.git > git init /home/hellxz/jenkins/home/workspace/demo # timeout=10 Fetching upstream changes from https://github.com/hellxz/cicd-demo.git > git --version # timeout=10 > git --version # 'git version 2.17.1' > git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10 Commit message: "update" > git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Build Artifact) [Pipeline] sh (maven building) + mvn clean package -DskipTests NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED [INFO] Scanning for projects... [INFO] [INFO] ----------------------< online.hellxz:cicd-demo >----------------------- [INFO] Building cicd-demo 0.0.1 [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo --- [INFO] [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 1 resource [INFO] Copying 0 resource [INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes [INFO] [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo --- [INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar [INFO] [INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.248 s [INFO] Finished at: 2020-12-23T11:51:20Z [INFO] ------------------------------------------------------------------------ [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Build Image) [Pipeline] sh (image building) + /bin/bash artifact2image.sh Sending build context to Docker daemon 17.26MB Step 1/5 : FROM openjdk:8-jdk ---> 89f100fa8f9f Step 2/5 : WORKDIR /root ---> Using cache ---> 17670bfa01f1 Step 3/5 : ADD target/*.jar app.jar ---> b7e4292e4f80 Step 4/5 : ENV JVM_OPTS="" ---> Running in 0be8c16c9c8a Removing intermediate container 0be8c16c9c8a ---> 3ef66b100082 Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom ${JVM_OPTS} -jar app.jar ---> Running in c8209a642a2d Removing intermediate container c8209a642a2d ---> bc7859f9b4fe Successfully built bc7859f9b4fe Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151 The push refers to repository [192.168.87.129:5000/cicd-demo] 44cbb90cc2c3: Preparing f85e383859a1: Preparing ffb4778f8a52: Preparing e528f2c31deb: Preparing c5f4367d4a59: Preparing ceecb62b2fcc: Preparing 193bc1d68b80: Preparing f0e10b20de19: Preparing ceecb62b2fcc: Waiting 193bc1d68b80: Waiting f0e10b20de19: Waiting ffb4778f8a52: Layer already exists e528f2c31deb: Layer already exists f85e383859a1: Layer already exists 193bc1d68b80: Layer already exists c5f4367d4a59: Layer already exists ceecb62b2fcc: Layer already exists f0e10b20de19: Layer already exists 44cbb90cc2c3: Pushed 20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006 Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151 Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291 Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4 Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9 Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951 [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Deploy k8s) [Pipeline] sh (deploy image to k8s) + /bin/bash deploy2k8s.sh deployment.apps/cicd-demo created service/cicd-demo-svc created [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] cleanWs [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] done [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS ``` 可以看到以下輸出，說明程式已部署到k8s叢集中了 ```bash deployment.apps/cicd-demo created service/cicd-demo-svc created ``` ssh登入k8s的機器驗證  ## 總結 簡單總結下，Jenkins與Kubernetes的整合，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通過 `構建並上傳映象` 和 `ssh傳送配置和命令` 實現的。 本文是參考慕課網上的[教程](https://www.imooc.com/learn/1112)，快速掃了下視訊實踐所得，是瞭解其思路後自行實現的。 實際操作上才發現，腦子說會了，然後雙手卻沒有跟上，還是要動手啊！ 操作了一天半左右，最開始還想著用vagrant初始化VM，由於工作上有個虛擬機器是VBox5的，升級新版不能用。搞了半天放棄了Vagrant（VBox6.1升級了命令引數不相容舊版，導致Vagrant為相容它把幾乎所有版本Vagrant都改了……）。 文中難免因為安裝部分的描述而過於冗長，感謝大家能看到這裡，最後再次感謝慕課網的[鹿哥](https://www.imooc.com/u/7711623/courses?sort=publish) ## 參考 https://www.imooc.com/learn/1112 https://cloud.tencent.com/developer/article/1394657 https://www.jianshu.com/p/3f13850d66ea https://www.cnblogs.com/hellxz/p/use-kubeadm-init-kubernetes-cluster.html https://www.cnblogs.com/hellxz/p/how-kubernetes-deploy-application.html https://developer.aliyun.com/article/221687?spm=a2c6h.12873639.0.0.755c6235EZuEht https://minikube.sigs.k8s.io/docs/start https://www.cnblogs.com/hellxz/p/install_jenkins.html https://www.cnblogs.com/hellxz/p/jenkins_install_plugins_fas