一、kubernetes故障排查

1、應用程式故障排查

①主要針對Pod級別的，非running狀態時使用describe檢視Pod事件進行問題排查。describe也可以檢視其他資源物件事件，如deployment、service等。

kubectl describe TYPE/NAME

[root@k8s-master ~]# kubectl describe pod web 
Name:         web
Namespace:    default
Priority:     0
Node:         k8s-node1/192.168.56.62
Start Time:   Wed, 
 
16 Dec 2020 14:43:55 +0800
Labels:       <none>
Annotations:  cni.projectcalico.org/podIP: 10.244.36.81/32
              cni.projectcalico.org/podIPs: 10.244.36.81/32
Status:       Pending
IP:           
IPs:          <none>
Containers:
  nginx:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           
 
80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c87dr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  
 
default-token-c87dr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-c87dr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                Message
  ----    ------     ----       ----                -------
  Normal  Scheduled  <unknown>  default-scheduler   Successfully assigned default/web to k8s-node1
  Normal  Pulling    11s        kubelet, k8s-node1  Pulling image "nginx"

kubectl logs TYPE/NAME [-c CONTAINER]：Apiserver呼叫kubelet的介面獲取

[root@k8s-master ~]# kubectl logs web 
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

kubectl exec POD [-c CONTAINER] --COMMAND [args...]，一個Pod中有多個容器時，使用-c指定容器的名稱。

②pod處於pending狀態可能的原因

• 下載映象
• 可能node節點資源不足
• 沒有匹配到節點標籤
• 有汙點

2、管理節點異常排查

叢集架構圖

①kubeadm部署

除kubelet服務外，其他元件均採用靜態Pod啟動。

②二進位制部署

所有元件均採用systemd管理

③管理節點元件

• kube-apiserver
• kube-controller-manager
• kube-scheduler