Warning  FailedMount            26s                kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63,auto_unmount,log-level=ERROR 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-7328.scope.
[2020-12-11 04:36:41.066237] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:41.097405] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:41.097454] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
  Normal   SuccessfulMountVolume  26s  kubelet, 10.145.132.62  MountVolume.SetUp succeeded for volume "default-token-t98sg"
  Warning  FailedMount            26s  kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o log-level=ERROR,log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63,auto_unmount 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-7707.scope.
[2020-12-11 04:36:41.801551] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:41.832556] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:41.832604] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
  Warning  FailedMount  24s  kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o auto_unmount,log-level=ERROR,log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-7830.scope.
[2020-12-11 04:36:43.009627] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:43.051426] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:43.051491] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
  Warning  FailedMount  22s  kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63,auto_unmount,log-level=ERROR 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-7970.scope.
[2020-12-11 04:36:45.202008] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:45.244826] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:45.244890] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
  Warning  FailedMount  18s  kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63,auto_unmount,log-level=ERROR 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-8172.scope.
[2020-12-11 04:36:49.395171] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:49.434848] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:49.434910] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
  Warning  FailedMount  10s  kubelet, 10.145.132.62  MountVolume.SetUp failed for volume "pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc --scope -- mount -t glusterfs -o log-file=/apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log,backup-volfile-servers=10.145.132.61:10.145.132.62:10.145.132.63,auto_unmount,log-level=ERROR 10.145.132.61:vol_c87e01813b64c4f30d3492b927b6c479 /apps/container/kubelet_data_dir/pods/7692949e-3b6a-11eb-a7ae-6c92bf8f6ffc/volumes/kubernetes.io~glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc
Output: Running scope as unit run-8422.scope.
[2020-12-11 04:36:57.591992] E [glusterfsd.c:828:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Check the log file /apps/container/kubelet_data_dir/plugins/kubernetes.io/glusterfs/pvc-1343ae7d-cd81-11ea-a7ae-6c92bf8f6ffc/gitlab-667bc689f4-kzdqs-glusterfs.log for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2020-12-11 04:36:57.638679] E [MSGID: 114044] [client-handshake.c:1031:client_setvolume_cbk] 0-vol_c87e01813b64c4f30d3492b927b6c479-client-1: SETVOLUME on remote-host failed: Authentication failed [許可權不夠]
[2020-12-11 04:36:57.638736] E [fuse-bridge.c:6358:notify] 0-fuse: Server authenication failed. Shutting down.
 

解決

因為glusterfs重啟了，導致掛載的時候報這樣的錯誤，重啟全部的glusterfs 和 heketi