rancher server偶發504問題排查
阿新 • • 發佈:2020-11-21
問題
rancher/server 部署到gce莫名其妙地出現504,導致訪問異常,看containers logs訪問網際網路都是失敗的
#docker logs [root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# journalctl -u docker -f -- Logs begin at Tue 2020-10-13 12:24:20 UTC. -- Nov 13 10:38:05 rancher dockerd[12162]: time="2020-11-13T10:38:05.335827262Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.158533175Z" level=info msg="Loading containers: done." Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.244847016Z" level=info msg="Docker daemon" commit=4484c46d9d graphdriver(s)=overlay2 version=19.03.13 Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.244930649Z" level=info msg="Daemon has completed initialization" Nov 13 10:38:06 rancher systemd[1]: Started Docker Application Container Engine. Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.297384575Z" level=info msg="API listen on /var/run/docker.sock" Nov 13 10:39:18 rancher dockerd[12162]: time="2020-11-13T10:39:18.271599362Z" level=info msg="Container 632ebda68cfbb44909669307112bb6e5e158ecaeb5677667be18320e948bf290 failed to exit within 10 seconds of signal 15 - using the force" Nov 13 10:39:18 rancher dockerd[12162]: time="2020-11-13T10:39:18.544146121Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Nov 19 03:54:02 rancher dockerd[12162]: time="2020-11-19T03:54:02.985747420Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Nov 19 03:54:28 rancher dockerd[12162]: time="2020-11-19T03:54:28.748323212Z" level=warning msg="IPv4 forwarding is disabled. Networking will not work." ^C [root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# ^C [root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# cat /proc/sys/net/ipv4/ip_forward 0
解決
cat > /etc/sysctl.d/docker.conf <<-'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
開啟網路轉發