交付dubbo 微服務到k8s 叢集
阿新 • • 發佈:2020-06-29
dubbo 微服務結構簡介
上圖中,藍色的表示與業務有互動,綠色的表示只對Dubbo內部互動。上述圖所描述的呼叫流程如下:
- 服務提供方釋出服務到服務註冊中心;
- 服務消費方從服務註冊中心訂閱服務;
- 服務消費方呼叫已經註冊的可用服務
1. 部署maven 並改成國內源
在 7.200 上: tar xf apache-maven-3.6.0-bin.tar.gz -C /data/nfs-volume/jenkins_home/maven-3.6.3
修改maven 倉庫為阿里雲倉庫
<mirrors> <mirror> <id>nexus-aliyun</id> <mirrorOf>*</mirrorOf> <name>Nexus aliyun</name> <url>http://maven.aliyun.com/nexus/content/groups/public</url> </mirror> </mirrors> #修改成阿里雲映象源
2. 準備 dubbo 底層映象
docker pull docker.io/stanleyws/jre8:8u112 #這個是jdk8的原始映象 docker tag docker.io/stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112 docker push harbor.od.com/public/jre:8u112 cat <<EOF >/data/dockerfile/jre8/Dockerfile FROM harbor.od.com/public/jre:8u112 RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo 'Asia/Shanghai' >/etc/timezone ADD config.yml /opt/prom/config.yml ADD jmx_javaagent-0.3.1.jar /opt/prom/ WORKDIR /opt/project_dir ADD entrypoint.sh /entrypoint.sh CMD ["/entrypoint.sh"] EOF cat <<EOF>config.yml --- rules: - pattern: '.*' EOF #下載java-agent 用來給普羅米修斯監控 wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar vim entrypoint.sh #!/bin/sh M_OPTS="-Duser.timezone=Asia/Shanghai-javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname-i):${M_PORT:-"12346"}:/opt/prom/config.yml" C_OPTS=${C_OPTS} JAR_BALL=${JAR_BALL} exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL} chmod +x entrypoint.sh #登入harbor 建立base 倉庫,許可權是public ]# docker build . -t harbor.od.com/base/jre8:8u112 && docker push harbor.od.com/base/jre8:8u112
3.交付dubbo server 到k8s 叢集
在 jenkins 中新建 一個pipline的專案,名字自定義,選擇引數構建
#然後到Jenkins新建pipline工程,在general,選擇thisprojectisparameterized。 選擇stringparameter。勾選trimisstring,分別是: app_nameimage_namegit_repogit_veradd_tagtarget_dir(./target)mvn_dir(./) mvn_cmd(mvncleanpackage-Dmaven.test.skip=true)#Choiceparameter base_image(base/jre8:8u112) maven(3.6.3)#執行編譯的軟體包版本,要根據實際mvn寫。
以此填入 自定義的引數,我這裡放個填好後的圖,就是 app_name image_name 那一列:
然後在jenkins pipeline 下面填入pipeline 指令碼
pipeline { agent any stages { stage('pull') { //get project code from repo steps { sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}" } } stage('build') { //exec mvn cmd steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}" } } stage('package') { //move jar file into project_dir steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir" } } stage('image') { //build image and push to registry steps { writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image} ADD ${params.target_dir}/project_dir /opt/project_dir"""#這一步是把 FROM harbor.od.com/${params.base_image} #ADD ${params.target_dir}/project_dir /opt/project_dir 具體內容寫到jenkins 裡面dockerfile 裡面 然後在jenkins 裡面構建有了新的jar 包的映象 並推送到 #habor 上,映象的名字字尾是剛才的add_tag 也就是時間 sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}" } } } }
儲存,之後再專案專案裡面選擇引數構建:
具體的引數如下:
git_repo 要填dubbo-server 的git 倉庫地址 ,git_ver 就是程式碼分支,這裡拉取的 apolo 分支的程式碼。
add_tag 建議用時間日期 base_image 就是上一步構建的dockerfile 基礎映象 maven 就是之前部署的,名字要和 7.200/data/nfs-volume/jenkins_home/ 裡面 maven 的目錄名字一致,要不jenkins 構建時候會找不到,類似如下報錯
+ /var/jenkins_home/maven-3.6.3/bin/mvn clean package -Dmaven.test.skip=true /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: 1: /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: /var/jenkins_home/maven-3.6.3/bin/mvn: not found
然後點選構建,成功後結果如下,這時候登陸harbor 就會發現多了 一個
app/dubbo-demo-service:apollo_20200517_1600 的映象了!
4. 準備dubbo server 的yaml 檔案
[root@hdss7-200 dubbo-server]# cat dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-service namespace: app labels: name: dubbo-demo-service spec: replicas: 1 selector: matchLabels: name: dubbo-demo-service template: metadata: labels: app: dubbo-demo-service name: dubbo-demo-service spec: containers: - name: dubbo-demo-service image: harbor.od.com/app/dubbo-demo-service:apollo_20200517_1600 #就是剛才新構建的映象 ports: - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: dubbo-server.jar - name: C_OPTS value: -Denv=dev -Dapollo.meta=http://config.od.com #這裡結合apolo 配置中心來做的 imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
5 交付dubbo-monitor 到k8s 叢集
#先下載dubbomonitor包 https://github.com/Jeromefromcn/dubbo-monitor#dubbo-monitor ,下載後解壓, 修改配置cat /opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner= dubbo.registry.address=zookeeper://10.4.7.11:2181 # 改成實際zk 地址 dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.log4j.file=logs/dubbo-monitor.log
準備dockerfile
vim /data/dockerfile/dubbo-monitor/Dockerfile FROM jeromefromcn/docker-alpine-java-bash MAINTAINER Jerome Jiang COPY dubbo-monitor-simple/ /dubbo-monitor-simple/ #dubbo-monitor-simple/ 就是dubbo-monitor 包裡面的資料夾 CMD /dubbo-monitor-simple/bin/start.sh
docker build . -t harbor.od.com/infra/dubbo-monitor:latest #構建映象
因為github 上 start.sh 指令碼在容器環境中啟動有問題 需要修改,下面是修改後的。
[root@hdss7-200 dubbo-monitor]# cat dubbo-monitor-simple/bin/start.sh #!/bin/bash sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties cd `dirname $0` BIN_DIR=`pwd` cd .. DEPLOY_DIR=`pwd` CONF_DIR=$DEPLOY_DIR/conf SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` SERVER_PROTOCOL=`sed '/dubbo.protocol.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` LOGS_FILE=`sed '/dubbo.log4j.file/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` if [ -z "$SERVER_NAME" ]; then SERVER_NAME=`hostname` fi PIDS=`ps -f | grep java | grep "$CONF_DIR" |awk '{print $2}'` if [ -n "$PIDS" ]; then echo "ERROR: The $SERVER_NAME already started!" echo "PID: $PIDS" exit 1 fi if [ -n "$SERVER_PORT" ]; then SERVER_PORT_COUNT=`netstat -tln | grep $SERVER_PORT | wc -l` if [ $SERVER_PORT_COUNT -gt 0 ]; then echo "ERROR: The $SERVER_NAME port $SERVER_PORT already used!" exit 1 fi fi LOGS_DIR="" if [ -n "$LOGS_FILE" ]; then LOGS_DIR=`dirname $LOGS_FILE` else LOGS_DIR=$DEPLOY_DIR/logs fi if [ ! -d $LOGS_DIR ]; then mkdir $LOGS_DIR fi STDOUT_FILE=$LOGS_DIR/stdout.log LIB_DIR=$DEPLOY_DIR/lib LIB_JARS=`ls $LIB_DIR|grep .jar|awk '{print "'$LIB_DIR'/"$0}'|tr "\n" ":"` JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true " JAVA_DEBUG_OPTS="" if [ "$1" = "debug" ]; then JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n " fi JAVA_JMX_OPTS="" if [ "$1" = "jmx" ]; then JAVA_JMX_OPTS=" -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false " fi JAVA_MEM_OPTS="" BITS=`java -version 2>&1 | grep -i 64-bit` if [ -n "$BITS" ]; then JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=32m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 " else JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC " fi echo -e "Starting the $SERVER_NAME ...\c" exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
6. 準備dobbo monitor 檔案
cat deployment.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra labels: name: dubbo-monitor spec: replicas: 1 selector: matchLabels: name: dubbo-monitor template: metadata: labels: app: dubbo-monitor name: dubbo-monitor spec: containers: - name: dubbo-monitor image: harbor.od.com/infra/dubbo-monitor:latest ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
[root@hdss7-200 dubbo-monitor]# cat cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: dubbo-monitor-cm namespace: infra data: dubbo.properties: | dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner=simplemonitor dubbo.registry.address=zookeeper://10.4.7.11:2181 #zk 的地址一定要和程式碼裡面保持一致! dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log dubbo.log4j.level=WARN [root@hdss7-200 dubbo-monitor]# cat ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra spec: rules: - host: dubbo-monitor.od.com http: paths: - path: / backend: serviceName: dubbo-monitor servicePort: 8080 [root@hdss7-200 dubbo-monitor]# cat svc.yaml kind: Service apiVersion: v1 metadata: name: dubbo-monitor namespace: infra spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-monitor
新增A 記錄 dubbo-monitor.od.com IN A 10.4.7.10 測試訪問 http://dubbo-monitor.od.com
7.交付 dubbo服務消費者到K8S 叢集
jenkins 上新構建一個pipeline 專案,新增的引數和dubbo-serve 一樣,,可以直接把dubbo-server 複製過來,然後填入引數 開始構建,引數如下圖:
看下 console output
Step 1/2 : FROM harbor.od.com/base/jre8:8u112 ---> 621a01f7f85a Step 2/2 : ADD ./dubbo-client/target/project_dir /opt/project_dir ---> 7d5577992ecd Successfully built 7d5577992ecd Successfully tagged harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 + docker push harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 The push refers to repository [harbor.od.com/app/dubbo-demo-consumer] 會看到harbor 上多了 app/dubbo-demo-consumer:apollo_20200517_1650 的映象
8. 準備consumer 的yaml 檔案
# dubbo-consumer 資源配置清單 dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-consumer namespace: app labels: name: dubbo-demo-consumer spec: replicas: 1 selector: matchLabels: name: dubbo-demo-consumer template: metadata: labels: app: dubbo-demo-consumer name: dubbo-demo-consumer spec: containers: - name: dubbo-demo-consumer image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 #就是剛才新構建的consumer 映象 ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: dubbo-client.jar imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 cat svc.yaml kind: Service apiVersion: v1 metadata: name: dubbo-demo-consumer namespace: app spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-demo-consumer cat ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-consumer namespace: app spec: rules: - host: demo.od.com http: paths: - path: / backend: serviceName: dubbo-demo-consumer servicePort: 8080
kubectl apply -f 這幾個檔案,然後 訪問 http://demo.od.com/hello?name=hello 看是否有返回結果,有返回就是正常的。
登入monitor 檢視: 可以看到 server 和 consumer 都有了。