1. 程式人生 > >優化:在k8s上部署的gitlab

優化:在k8s上部署的gitlab

### gitlab元件圖 ![](https://img2020.cnblogs.com/blog/1464583/202007/1464583-20200704171103632-1903779866.png) ### gitlab在k8s上佔用資源 ```yaml # kubectl top pods -n default | grep git* gitlab-gitaly-0 9m 444Mi gitlab-gitlab-exporter-59c6bdb69c-gz9nf 5m 36Mi gitlab-gitlab-shell-547cc48d99-kmrbp 10m 21Mi gitlab-gitlab-shell-547cc48d99-szg74 8m 25Mi gitlab-minio-5746f7f7c7-tdff5 0m 47Mi gitlab-prometheus-server-7d8bcf896d-vm5kq 14m 650Mi gitlab-redis-cd6b45457-2s4cg 9m 17Mi gitlab-registry-6c5df4646f-mbdkr 0m 15Mi gitlab-registry-6c5df4646f-mqkxl 0m 31Mi gitlab-sidekiq-all-in-1-576d89544d-8swmw 28m 967Mi gitlab-unicorn-68f8f8d8d9-6clpv 8m 1510Mi gitlab-unicorn-68f8f8d8d9-xbt8j 10m 1522Mi ``` ### gitlab元件簡介 對比kubectl top pods -n default | grep git* 輸出 1. gitlab-gitaly:Gitaly是一項由GitLab設計的服務,旨在消除在GitLab的分散式部署(請考慮GitLab.com或高可用性部署)中對Git儲存使用NFS的需求。從11.3.0開始,此服務處理GitLab中的所有Git級別訪問。 2. gitlab-gitlab-exporter:GitLab Exporter是一個內部設計的流程,允許我們將有關GitLab應用程式內部的度量匯出到Prometheus。可以[在專案的README中](https://gitlab.com/gitlab-org/gitlab-exporter)閱讀更多內容。 3. gitlab-gitlab-shell:[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell)是[GitLab](https://gitlab.com/gitlab-org/gitlab-shell)設計的程式,用於處理基於SSH的`git`會話,並修改授權金鑰的列表。GitLab Shell不是Unix Shell,也不是Bash或Zsh的替代品。 4. gitlab-minio:MinIO是根據Apache License v2.0釋出的物件儲存伺服器。它與Amazon S3雲端儲存服務相容。它最適合儲存非結構化資料,例如照片,視訊,日誌檔案,備份和容器/ VM映像。一個物件的大小範圍可以從幾個KB到最大5TB。 5. gitlab-prometheus-server:Prometheus是一個時序工具,可幫助GitLab管理員公開有關用於向GitLab提供服務的各個流程的指標。 6. gitlab-redis:鍵值儲存/非持久資料庫。 7. gitlab-registry:gitlab自帶映象倉庫 8. gitlab-sidekiq:Sidekiq是Ruby後臺作業處理器,可從Redis佇列中提取作業並進行處理。後臺作業使GitLab通過將工作移至後臺來提供更快的請求/響應週期。 9. gitlab-unicorn:[Unicorn](https://yhbt.net/unicorn/)是一個Ruby應用程式伺服器,用於執行核心的Rails應用程式,該應用程式在GitLab中提供面向使用者的功能。通常,過程輸出,你會認為這`bundle`還是`config.ru`取決於GitLab版本。 ### 疑問點 ![image.png](https://cdn.nlark.com/yuque/0/2020/png/1143489/1593846191330-5bc666db-b5f1-42e6-8c76-6c0423c4955c.png) ### 目的 對在k8s上部署的gitlab進行記憶體優化 思路: \1. 尋找限制記憶體的方法((1)gitlab自身的資源限制策略【最好】,k8s做資源限制【最差】) 2.刪除非必要的應用程式(如刪除prometheus【後期會把gitlab遷移到裸機並且對這個集群系統做監控】)。 ### 策略 #### 嘗試關閉prometheus元件,看是否對gitlab有影響 檢視與gitlab-prometheus有關的資源清單 ```yaml # kubectl get svc,pod,ingress,deployment | grep prom ``` 發現沒有prometheus的ingress,這說明沒有走ingress-nginx代理,也能說明沒有人曾訪問過!!! 匯出gitlab_prometheus_deployment.yaml ```yaml # kubectl get deployment gitlab-prometheus-server -o yaml > gitlab_prometheus_deployment.yaml # cat gitlab_prometheus_deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" creationTimestamp: "2019-10-21T04:14:50Z" generation: 1 labels: app: prometheus chart: prometheus-5.5.3 component: server heritage: Tiller release: gitlab name: gitlab-prometheus-server namespace: default resourceVersion: "1438304832" selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-prometheus-server uid: 5360e3bf-f3b9-11e9-800a-8ad2fc7e2fa8 spec: progressDeadlineSeconds: 2147483647 replicas: 1 revisionHistoryLimit: 2147483647 selector: matchLabels: app: prometheus component: server release: gitlab strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: app: prometheus component: server release: gitlab spec: containers: - args: - --volume-dir=/etc/config - --webhook-url=http://localhost:9090/-/reload image: jimmidyson/configmap-reload:v0.1 imagePullPolicy: IfNotPresent name: prometheus-server-configmap-reload resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/config name: config-volume readOnly: true - args: - --storage.tsdb.retention=15d - --config.file=/etc/config/prometheus.yml - --storage.tsdb.path=/data - --web.console.libraries=/etc/prometheus/console_libraries - --web.console.templates=/etc/prometheus/consoles - --web.enable-lifecycle image: prom/prometheus:v2.2.1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /-/healthy port: 9090 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 name: prometheus-server ports: - containerPort: 9090 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /-/ready port: 9090 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/config name: config-volume - mountPath: /data name: storage-volume dnsPolicy: ClusterFirst initContainers: - command: - chown - -R - 65534:65534 - /data image: busybox:latest imagePullPolicy: IfNotPresent name: init-chown-data resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /data name: storage-volume restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: gitlab-prometheus-server serviceAccountName: gitlab-prometheus-server terminationGracePeriodSeconds: 300 volumes: - configMap: defaultMode: 420 name: gitlab-prometheus-server name: config-volume - name: storage-volume persistentVolumeClaim: claimName: gitlab-prometheus-server ``` 匯出gitlab_prometheus_svc.yaml ```yaml # kubectl get svc gitlab-prometheus-server -o yaml > gitlab_prometheus_svc.yaml # cat gitlab_prometheus_svc.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: "2019-10-21T04:14:50Z" labels: app: prometheus chart: prometheus-5.5.3 component: server heritage: Tiller release: gitlab name: gitlab-prometheus-server namespace: default resourceVersion: "144804734" selfLink: /api/v1/namespaces/default/services/gitlab-prometheus-server uid: 535378f3-f3b9-11e9-800a-8ad2fc7e2fa8 spec: clusterIP: 192.168.247.105 ports: - name: http port: 80 protocol: TCP targetPort: 9090 selector: app: prometheus component: server release: gitlab sessionAffinity: None type: ClusterIP ``` 大膽刪除deployment和svc,能有啥事? ```yaml # kubectl delete -f ``` 手動測試,似乎沒得啥問題。 匯出gitlab_exporter_deployment.yaml ```yaml # kubectl get deployment gitlab-gitlab-exporter -o yaml > gitlab_exporter_deployment.yaml # cat gitlab_exporter_deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "2" creationTimestamp: "2019-10-21T04:14:50Z" generation: 2 labels: app: gitlab-exporter chart: gitlab-exporter-2.4.6 heritage: Tiller release: gitlab name: gitlab-gitlab-exporter namespace: default resourceVersion: "1399100783" selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-gitlab-exporter uid: 5358e5e4-f3b9-11e9-800a-8ad2fc7e2fa8 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: gitlab-exporter release: gitlab strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: checksum/config: 158ed2dba0a2fbaa0d3fda906d70cf0eb7a80650e8f01d3046aa9a6fa0bba59a prometheus.io/path: /metrics prometheus.io/port: "9168" prometheus.io/scrape: "true" creationTimestamp: null labels: app: gitlab-exporter release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: gitlab-exporter release: gitlab topologyKey: kubernetes.io/hostname weight: 1 containers: - env: - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab-exporter/templates - name: CONFIG_DIRECTORY value: /etc/gitlab-exporter image: registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter:5.0.1 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /bin/bash - -c - pkill -f 'gitlab-exporter' livenessProbe: exec: command: - pgrep - -f - gitlab-exporter failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: gitlab-exporter ports: - containerPort: 9168 name: gitlab-exporter protocol: TCP readinessProbe: exec: command: - pgrep - -f - gitlab-exporter failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 50m memory: 100M terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/opt/gitlab-exporter/templates/gitlab-exporter.yml.erb name: gitlab-exporter-config subPath: gitlab-exporter.yml.erb - mountPath: /etc/gitlab name: gitlab-exporter-secrets readOnly: true - mountPath: /etc/ssl/certs/ name: etc-ssl-certs readOnly: true dnsPolicy: ClusterFirst initContainers: - image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3 imagePullPolicy: IfNotPresent name: certificates resources: requests: cpu: 50m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/ssl/certs name: etc-ssl-certs - command: - sh - /config/configure image: busybox:latest imagePullPolicy: Always name: configure resources: requests: cpu: 50m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /config name: gitlab-exporter-config readOnly: true - mountPath: /init-config name: init-gitlab-exporter-secrets readOnly: true - mountPath: /init-secrets name: gitlab-exporter-secrets restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1000 runAsUser: 1000 terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: gitlab-gitlab-exporter name: gitlab-exporter-config - name: init-gitlab-exporter-secrets projected: defaultMode: 256 sources: - secret: items: - key: postgres-password path: postgres/psql-password name: gitlab-postgresql-password - secret: items: - key: secret path: redis/password name: gitlab-redis-secret - emptyDir: medium: Memory name: gitlab-exporter-secrets - emptyDir: medium: Memory name: etc-ssl-certs ``` 資源清單好長,有點不想看 emmmmm 還是分析一波吧 看過後,刪除exporter的資源清單 ```yaml # kubectl delete -f gitlab_exporter_deployment.yaml deployment.extensions "gitlab-gitlab-exporter" deleted ``` 匯出gitlab-exporter的svc資源清單 ```yaml # kubectl get svc gitlab-gitlab-exporter -o yaml > gitlab_exporter_svc.yaml [root@mobanji gitlab]# cat gitlab_exporter_svc.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: "2019-10-21T04:14:50Z" labels: app: gitlab-exporter chart: gitlab-exporter-2.4.6 heritage: Tiller release: gitlab name: gitlab-gitlab-exporter namespace: default resourceVersion: "219393250" selfLink: /api/v1/namespaces/default/services/gitlab-gitlab-exporter uid: 534c0460-f3b9-11e9-800a-8ad2fc7e2fa8 spec: clusterIP: 192.168.180.175 ports: - name: gitlab-exporter port: 9168 protocol: TCP targetPort: 9168 selector: app: gitlab-exporter release: gitlab sessionAffinity: None type: ClusterIP ``` 分析一下,沒發現重要資訊,也刪了吧 ```yaml # kubectl delete -f gitlab_exporter_svc.yaml service "gitlab-gitlab-exporter" deleted ``` 至此,與prometheus有關的資源被從kubernetes上刪除。 測試驗證。1. 手動重新整理gitlab介面看有問題沒 2. 使用gitlab釋出一個臨時專案看有問題沒。 經過驗證暫時沒的問題 #### 限制gitlab unicorn記憶體使用 目前gitlab在k8s上的資源佔用情況 ```yaml # kubectl top pods | grep gitlab gitlab-gitaly-0 9m 366Mi gitlab-gitlab-shell-547cc48d99-kmrbp 18m 19Mi gitlab-gitlab-shell-547cc48d99-szg74 15m 17Mi gitlab-minio-5746f7f7c7-tdff5 0m 41Mi gitlab-redis-cd6b45457-2s4cg 9m 14Mi gitlab-registry-6c5df4646f-mbdkr 0m 31Mi gitlab-registry-6c5df4646f-mqkxl 0m 40Mi gitlab-sidekiq-all-in-1-576d89544d-8swmw 28m 967Mi gitlab-unicorn-68f8f8d8d9-6clpv 7m 1514Mi gitlab-unicorn-68f8f8d8d9-xbt8j 8m 1526Mi ``` 可以看出 目前佔用資源最高的是unicorn元件。Unicorn]是一個Ruby應用程式伺服器,用於執行核心的Rails應用程式,該應用程式在GitLab中提供面向使用者的功能。通常,過程輸出,你會認為這`bundle`還是`config.ru`取決於GitLab版本。 通過官網可以看出,這元件屬於核心元件。貌似還不能動。還是先匯出deployment,然後根據資源清單進行判斷吧 ```yaml # kubectl get deployment gitlab-unicorn -o yaml > gitlab_unicorn_deployment.yaml # cat gitlab_unicorn_deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "2" creationTimestamp: "2019-10-21T04:14:50Z" generation: 16 labels: app: unicorn chart: unicorn-2.4.6 heritage: Tiller release: gitlab name: gitlab-unicorn namespace: default resourceVersion: "1359934803" selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-unicorn uid: 535e0ed5-f3b9-11e9-800a-8ad2fc7e2fa8 spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: unicorn release: gitlab strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: checksum/config: 2d845854ae56029b9ee0a30ca793e5c0f74c3d708329ba605bbdc36000023517 cluster-autoscaler.kubernetes.io/safe-to-evict: "true" prometheus.io/path: /-/metrics prometheus.io/port: "8080" prometheus.io/scrape: "true" creationTimestamp: null labels: app: unicorn release: gitlab spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: unicorn release: gitlab topologyKey: kubernetes.io/hostname weight: 1 containers: - env: - name: TMPDIR value: /tmp/gitlab - name: GITALY_FEATURE_DEFAULT_ON value: "1" - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: prometheus_multiproc_dir value: /metrics image: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ee:v12.4.2 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /bin/bash - -c - pkill -SIGQUIT -f 'unicorn master' livenessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 initialDelaySeconds: 20 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 30 name: unicorn ports: - containerPort: 8080 name: unicorn protocol: TCP readinessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: requests: cpu: 300m memory: 1200M #有requests,沒有limits。只做最低保障,不做硬限制。還不能貿然做limits限制 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /metrics name: unicorn-metrics - mountPath: /var/opt/gitlab/templates name: unicorn-config - mountPath: /etc/gitlab name: unicorn-secrets readOnly: true - mountPath: /srv/gitlab/config/secrets.yml name: unicorn-secrets subPath: rails-secrets/secrets.yml - mountPath: /srv/gitlab/config/initializers/smtp_settings.rb name: unicorn-config subPath: smtp_settings.rb - mountPath: /srv/gitlab/INSTALLATION_TYPE name: unicorn-config subPath: installation_type - mountPath: /srv/gitlab/public/uploads/tmp name: shared-upload-directory - mountPath: /srv/gitlab/shared name: shared-artifact-directory - mountPath: /tmp name: shared-tmp - mountPath: /etc/ssl/certs/ name: etc-ssl-certs readOnly: true - env: - name: TMPDIR value: /tmp/gitlab - name: GITLAB_WORKHORSE_EXTRA_ARGS - name: GITLAB_WORKHORSE_LISTEN_PORT value: "8181" - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v12.4.2 imagePullPolicy: IfNotPresent livenessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 initialDelaySeconds: 20 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 30 name: gitlab-workhorse ports: - containerPort: 8181 name: workhorse protocol: TCP readinessProbe: exec: command: - /scripts/healthcheck failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: requests: cpu: 100m memory: 100M terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/opt/gitlab/templates name: workhorse-config - mountPath: /etc/gitlab name: workhorse-secrets readOnly: true - mountPath: /srv/gitlab/public/uploads/tmp name: shared-upload-directory - mountPath: /srv/gitlab/shared name: shared-artifact-directory - mountPath: /tmp name: shared-tmp - mountPath: /etc/ssl/certs/ name: etc-ssl-certs readOnly: true dnsPolicy: ClusterFirst initContainers: - image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3 imagePullPolicy: IfNotPresent name: certificates resources: requests: cpu: 50m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/ssl/certs name: etc-ssl-certs - args: - -c - sh -x /config-unicorn/configure ; sh -x /config-workhorse/configure ; mkdir -p -m 3770 /tmp/gitlab command: - sh image: busybox:latest imagePullPolicy: Always name: configure resources: requests: cpu: 50m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /config-unicorn name: unicorn-config readOnly: true - mountPath: /config-workhorse name: workhorse-config readOnly: true - mountPath: /init-config name: init-unicorn-secrets readOnly: true - mountPath: /init-secrets name: unicorn-secrets - mountPath: /init-secrets-workhorse name: workhorse-secrets - mountPath: /tmp name: shared-tmp - args: - /scripts/wait-for-deps env: - name: GITALY_FEATURE_DEFAULT_ON value: "1" - name: CONFIG_TEMPLATE_DIRECTORY value: /var/opt/gitlab/templates - name: CONFIG_DIRECTORY value: /srv/gitlab/config - name: WORKHORSE_ARCHIVE_CACHE_DISABLED value: "1" image: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ee:v12.4.2 imagePullPolicy: IfNotPresent name: dependencies resources: requests: cpu: 50m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/opt/gitlab/templates name: unicorn-config - mountPath: /etc/gitlab name: unicorn-secrets readOnly: true restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1000 runAsUser: 1000 terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: shared-tmp - emptyDir: medium: Memory name: unicorn-metrics - configMap: defaultMode: 420 name: gitlab-unicorn name: unicorn-config - configMap: defaultMode: 420 name: gitlab-workhorse-config name: workhorse-config - name: init-unicorn-secrets projected: defaultMode: 256 sources: - secret: items: - key: secrets.yml path: rails-secrets/secrets.yml name: gitlab-rails-secret - secret: items: - key: secret path: shell/.gitlab_shell_secret name: gitlab-gitlab-shell-secret - secret: items: - key: token path: gitaly/gitaly_token name: gitlab-gitaly-secret - secret: items: - key: secret path: redis/password name: gitlab-redis-secret - secret: items: - key: postgres-password path: postgres/psql-password name: gitlab-postgresql-password - secret: items: - key: registry-auth.key path: registry/gitlab-registry.key name: gitlab-registry-secret - secret: items: - key: shared_secret path: gitlab-workhorse/secret name: gitlab-gitlab-workhorse-secret - secret: items: - key: accesskey path: minio/accesskey - key: secretkey path: minio/secretkey name: gitlab-minio-secret - secret: items: - key: password path: smtp/smtp-password name: smtp-password - emptyDir: medium: Memory name: unicorn-secrets - emptyDir: medium: Memory name: workhorse-secrets - emptyDir: {} name: shared-upload-directory - emptyDir: {} name: shared-artifact-directory - emptyDir: medium: Memory name: etc-ssl-certs ``` 得了,分析了一波還是決定不動了,優化1G也算