Jenkins在K8s的pod中實現docker in docker 構建映象併發布到K8s叢集中
阿新 • • 發佈：2021-06-18
一、執行架構
Jenkins的kubernetes plugin在執行構建時會在kubernetes叢集中自動建立一個Pod，並在Pod內部建立一個名為jnlp的容器，該容器會連線Jenkins並執行Agent程式，形成一個Jenkins的Master和Slave架構，然後Slave會執行構建指令碼進行構建。
二、部署指令碼
jenkins連線K8s叢集可參考此方法：https://www.cnblogs.com/cyleon/p/14893198.html
2.1 node前端專案編譯部署指令碼pipeline指令碼
pipeline { agent { kubernetes { yaml""" kind: Pod metadata: name: jenkins-agent namespace: default spec: initContainers: - name: init image: busybox:latest imagePullPolicy: IfNotPresent securityContext: privileged: true command: ["sh", "-c", "chown -R 1000:1000 /home/jenkins/agent/workspace"] volumeMounts:- name: hostpathjenkins mountPath: "/home/jenkins/agent/workspace" containers: - name: jnlp image: ccr.ccs.tencentyun.com/jenkins-agent:4.3-4 imagePullPolicy: IfNotPresent resources: requests: cpu: 100m memory: 256Mi volumeMounts: - name: hostpathjenkins mountPath:"/home/jenkins/agent/workspace" - name: node image: ccr.ccs.tencentyun.com/jenkins-node:v1 imagePullPolicy: IfNotPresent command: - cat tty: true resources: requests: cpu: 100m memory: 512Mi limits: cpu: 2000m memory: 3072Mi volumeMounts: - name: hostpathjenkins mountPath: "/home/jenkins/agent/workspace" - name: ansible image: ccr.ccs.tencentyun.com/ansible:centos7 imagePullPolicy: IfNotPresent command: - cat tty: true volumeMounts: - name: nfs-volume mountPath: /data - name: hostpathjenkins mountPath: "/home/jenkins/agent/workspace" restartPolicy: Never nodeSelector: jenkins: enable imagePullSecrets: - name: docker-registry volumes: - name: hostpathjenkins hostPath: path: /var/lib/kubelet/jenkins - name: nfs-volume nfs: server: 192.168.7.77 path: "/eo97869g" """ } } environment{ CRED_ID='98ee5839-c832-4b64-92cf-77effbeeeb64' GITLAB_URL="https://git.lzfn.cn/FE/fe-test.git" SERVER_GROUP="192.168.7.80" PROJECT_NAME="${JOB_BASE_NAME}" ANSIBLE_FILE="/data/script/k8s/yaml/main_static.yaml" } options { disableConcurrentBuilds() timeout(time: 1, unit: 'HOURS') buildDiscarder logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '15') timestamps () } // 引數化構建 parameters { choice choices: ['NO', 'YES'], description: '是否執行 npm install ?', name: 'npm_install' choice choices: ['node-v10.10.0', 'node-v11.10.0', 'node-v12.10.0', 'node-v13.10.0', 'node-v14.10.0'], description: '請選擇node版本', name: 'npm_version' choice choices: ['https://registry.npm.taobao.org'], description: '請選擇 npm 的 registry', name: 'npm_registry' choice choices: ['npm run build:test', 'npm run build', 'npm run build:h5', 'npm run docs', 'npm run test', 'npm run build:prod'], description: '請選擇編譯引數', name: 'npm_run_build' gitParameter name: 'branch', branchFilter: 'origin/(.*)', defaultValue: 'master', listSize: '7', quickFilterEnabled: true, selectedValue: 'TOP', sortMode: 'DESCENDING_SMART', tagFilter: '*', type: 'PT_BRANCH' } stages { stage('程式碼下載') { steps { wrap([$class: 'BuildUser']) { script { currentBuild.description = "<p><font color='red'>branch: ${params.branch}</font></p><p><font color='green'>npm_version: ${npm_version}</p><p>釋出人員: ${BUILD_USER}</p>" } } checkout( [ $class: 'GitSCM', branches: [[name: "${params.branch}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], gitTool: 'Default', userRemoteConfigs: [ [ credentialsId: "${CRED_ID}", url: "${GITLAB_URL}" ] ] ] ) } } stage('安裝依賴') { when { equals expected: "YES", actual: "${npm_install}" } steps { container('node') { sh ''' export NODE_HOME=/usr/local/${npm_version}-linux-x64 export PATH=$PATH:$NODE_HOME/bin rm -rf node_modules ln -sv /usr/local/${npm_version}-linux-x64/bin/node /usr/bin/node /usr/local/${npm_version}-linux-x64/bin/npm install -g cnpm --registry=https://registry.npm.taobao.org /usr/local/${npm_version}-linux-x64/bin/cnpm config set registry ${npm_registry} /usr/local/${npm_version}-linux-x64/bin/cnpm get registry /usr/local/${npm_version}-linux-x64/bin/cnpm install ''' } } } stage('程式碼打包') { steps { container('node') { sh ''' export NODE_HOME=/usr/local/${npm_version}-linux-x64 export PATH=$PATH:$NODE_HOME/bin /usr/local/${npm_version}-linux-x64/bin/${npm_run_build} ''' } } } stage('程式碼部署'){ steps{ container('ansible') { ansiColor('xterm') { ansiblePlaybook( playbook: "${ANSIBLE_FILE}", colorized: true, extraVars: [ project: [value: "${PROJECT_NAME}", hidden: false], server_group: [value: "${SERVER_GROUP}", hidden: false], project_group: [value: "${JOB_BASE_NAME}", hidden: false], workspace: [value: "${env.WORKSPACE}", hidden: false], ] ) } } } } } }
2.2 配置好Jenkins釋出頁面選項頁面如下：
2.3 點選開始構建，構建過程如下：
2.4 構建的時候會在K8s叢集內建立一個pod，其中包含3個容器，構建完成後會銷燬pod。
三、製作映象構建Dockerfile
jnlp 容器：用來連線jenkins master，直接使用官方的映象即可；jenkins/inbound-agent:4.3-4
node容器：裡面拷貝了多個node編譯用的的二進位制包
ansible容器：編譯完成之後將程式碼推送到目標機器
3.1 node映象
我是將node包提前下載到本地了，沒有寫在指令碼中，經測試直接在容器中下載會使映象稍大一點，提供兩個下載地址
https://nodejs.org/dist/v10.10.0/node-v10.10.0-linux-x64.tar.xz
https://mirror.tuna.tsinghua.edu.cn/nodejs-release/v10.10.0/node-v10.10.0-linux-x64.tar.xz
# cat Dockerfile FROM centos:7 MAINTAINER 532141928@qq.com RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo 'Asia/Shanghai' >/etc/timezone ADD node-v10.10.0-linux-x64.tar.xz node-v11.10.0-linux-x64.tar.xz node-v12.10.0-linux-x64.tar.xz node-v13.10.0-linux-x64.tar.xz node-v14.10.0-linux-x64.tar.xz /usr/local/CMD ["cat"]
3.2 ansible映象
# cat Dockerfile FROM centos:7 MAINTAINER lzfn@qq.com RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo 'Asia/Shanghai' >/etc/timezone \ && yum install epel-release -y \ && yum install ansible -y \ && yum clean all \ && mkdir /root/.ssh COPY id_rsa /root/.ssh/id_rsa COPY hosts /etc/ansible/hosts RUN chmod 700 /root/.ssh \ && chmod 600 /root/.ssh/id_rsa \ && sed -i 's/#host_key_checking = False/host_key_checking = False/g' /etc/ansible/ansible.cfg CMD ["cat"] # cat hosts [test-fe] 192.168.7.80 ansible_port=22 ansible_ssh_user=lzfn ansible_become=yes
四、ansible指令碼如下
# cat main_static.yaml - hosts: "{{ server_group }}" vars: DIST_PATH: "{{ workspace }}" tasks: - name: Finding static dir shell: "find {{ workspace }}/dist/ -type d -name static" ignore_errors: True connection: local register: STATIC - name: Create Folder file: path={{ item }} state=directory owner=www group=www with_items: - '/data/wwwroot/{{ project_group }}/{{project}}/' - '/data/tmp/{{ project_group }}/{{project}}/' - name: create static dir file: path={{ item }} state=directory owner=www group=www with_items: - '/data/wwwroot/{{ project_group }}/{{project}}/dist/static/' when: STATIC.stdout != "" - name: check directory exists local_action: stat path={{ DIST_PATH }} register: dist_path - fail: msg="not found dist" when: not dist_path.stat.exists and not dist_path.stat.isdir - name: Rsync dist file copy: src: "{{ DIST_PATH }}/dist/" dest: "/data/tmp/{{ project_group }}/{{ project }}/dist/" owner: www group: www mode: '0644' - name: increment to static dir shell: "rsync -a /data/tmp/{{ project_group }}/{{ project }}/dist/static/ /data/wwwroot/{{ project_group }}/{{ project }}/dist/static/" when: STATIC.stdout != "" - name: Cover Code to Product Directory shell: "rsync -a --exclude 'static' --delete /data/tmp/{{ project_group }}/{{ project }}/ /data/wwwroot/{{ project_group }}/{{ project }}/"