2. 程式人生 > >[k8s]k8s 1.9(on the fly搭建) 1.9_cni-flannel部署排錯 ipvs模式

[k8s]k8s 1.9(on the fly搭建) 1.9_cni-flannel部署排錯 ipvs模式

阿新 • • 發佈：2018-01-01
wap sch byte cer ica 同步 disable cred 0.12

角色 節點名 節點ip
master n1 192.168.14.11
節點1 n2 192.168.14.12
節點2 n3 192.168.14.13

https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/

技術分享圖片

準備token.csv和bootstrap.kubeconfig文件

- 在master生成token.csv
BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"
cat > token.csv<<EOF
41f7e4ba8b7be874fcff18bf5cf41a7c,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

- 將bootstrap.kubeconfig同步到所有節點
設置集群參數
kubectl config set-cluster kubernetes   --certificate-authority=/root/ssl/ca.crt   --embed-certs=true   --server=http://192.168.14.11:8080   --kubeconfig=bootstrap.kubeconfig
  
設置客戶端認證參數
kubectl config set-credentials kubelet-bootstrap   --token="41f7e4ba8b7be874fcff18bf5cf41a7c"   --kubeconfig=bootstrap.kubeconfig

設置上下文參數
kubectl config set-context default   --cluster=kubernetes   --user=kubelet-bootstrap   --kubeconfig=bootstrap.kubeconfig

設置默認上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

兩個文件我都放在了/root下.

k8s 1.9 on the fly啟動

etcd --advertise-client-urls=http://192.168.14.11:2379 --listen-client-urls=http://0.0.0.0:2379 --debug

kube-apiserver --service-cluster-ip-range=10.254.0.0/16 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --storage-backend=etcd2 --v=2 --enable-bootstrap-token-auth --token-auth-file=/root/token.csv


kube-controller-manager   --master=http://127.0.0.1:8080   --service-account-private-key-file=/root/ssl/ca.key  --cluster-signing-cert-file=/root/ssl/ca.crt --cluster-signing-key-file=/root/ssl/ca.key --root-ca-file=/root/ssl/ca.crt --v=2 
  
kube-scheduler --master=http://127.0.0.1:8080 --v=2 


kubelet --allow-privileged=true --cluster-dns=10.254.0.2 --cluster-domain=cluster.local --v=2 --experimental-bootstrap-kubeconfig=/root/bootstrap.kubeconfig --kubeconfig=/root/kubelet.kubeconfig --fail-swap-on=false


kube-proxy  --master=http://192.168.14.11:8080  --v=2


kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

api相對1.7的變化:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#before-upgrading
https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

  • kubelet沒了--api-servers參數,必須用bootstrap方式去連api
  • 移除了 --runtime-config=rbac.authorization.k8s.io/v1beta1 配置,因為 RBAC 已經穩定，被納入了 v1 api，不再需要指定開啟
  • --authorization-mode 授權模型增加了 Node 參數，因為 1.8 後默認 system:node role 不會自動授予 system:nodes 組
  • 增加 --audit-policy-file 參數用於指定高級審計配置
  • 移除 --experimental-bootstrap-token-auth 參數，更換為 --enable-bootstrap-token-auth

k8s1.9 cni(flannel) on the fly啟動

註: HostPort不能使用CNI網絡插件(docker run -p 8081:8080)。這意味著pod中所有HostPort屬性將被簡單地忽略。

mkdir -p /etc/cni/net.d /opt/cni/bin
wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz
tar xf cni-plugins-amd64-v0.6.0.tgz -C /opt/cni/bin


cat > /etc/cni/net.d/10-flannel.conflist<<EOF
{
  "name":"cni0",
  "cniVersion":"0.3.1",
  "plugins":[
    {
      "type":"flannel",
      "delegate":{
        "forceAddress":true,
        "isDefaultGateway":true
      }
    },
    {
      "type":"portmap",
      "capabilities":{
        "portMappings":true
      }
    }
  ]
}
EOF
etcd --advertise-client-urls=http://192.168.14.11:2379 --listen-client-urls=http://0.0.0.0:2379 --debug


kube-apiserver --service-cluster-ip-range=10.254.0.0/16 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --storage-backend=etcd2 --v=2 --enable-bootstrap-token-auth --token-auth-file=/root/token.csv


kube-controller-manager   --master=http://127.0.0.1:8080   --service-account-private-key-file=/root/ssl/ca.key  --cluster-signing-cert-file=/root/ssl/ca.crt --cluster-signing-key-file=/root/ssl/ca.key --root-ca-file=/root/ssl/ca.crt --v=2  --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16


kube-scheduler --master=http://127.0.0.1:8080 --v=2 


kubelet --allow-privileged=true --cluster-dns=10.254.0.2 --cluster-domain=cluster.local --v=2 --experimental-bootstrap-kubeconfig=/root/bootstrap.kubeconfig --kubeconfig=/root/kubelet.kubeconfig --fail-swap-on=false --network-plugin=cni


kube-proxy  --master=http://192.168.14.11:8080  --v=2
 kubectl apply -f https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/cni-flannel.yml

查看pod確實從cni0分到了地址

[root@n1 ~]# kk
NAMESPACE     NAME                 READY     STATUS    RESTARTS   AGE       IP              NODE        LABELS
default       b1                   1/1       Running   0          6m        10.244.0.2      n2.ma.com   <none>
default       b2                   1/1       Running   0          6m        10.244.1.2      n3.ma.com   <none>

[root@n1 ~]# kubectl exec -it b1 sh
/ # ping 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=62 time=6.292 ms
64 bytes from 10.244.1.2: seq=1 ttl=62 time=0.981 ms

遇到的報錯

kubectl apply -f cni-flannel.yml沒提示報錯,但get pod無顯示

kubectl apply -f https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/cni-flannel.yml


原因: yaml用到了sa未創建
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system

pod0無cni0網卡,創建pod後分到的是172.17.x.x即docker0分配的地址

原因: kubelet未加cni啟動參數
--network-plugin=cni

創建成功後提示.kubectl create成功後一直pending,kubelet報錯

因為ipv6沒開,參考how-to-disable-ipv6

ifconfig -a | grep inet6
I1231 23:22:08.343188   15369 kubelet.go:1881] SyncLoop (PLEG): "busybox_default(5a16fa0c-ee3e-11e7-9220-000c29bfdc52)", event: &pleg.PodLifecycleEvent{ID:"5a16fa0c-ee3e-11e7-9220-000c29bfdc52", Type:"ContainerDied", Data:"76e584c83f8dd3d54c759ac637bb47aa2a94de524372e282abde004d1cfbcd1b"}
W1231 23:22:08.343338   15369 pod_container_deletor.go:77] Container "76e584c83f8dd3d54c759ac637bb47aa2a94de524372e282abde004d1cfbcd1b" not found in pod's containers
I1231 23:22:08.644401   15369 kuberuntime_manager.go:403] No ready sandbox for pod "busybox_default(5a16fa0c-ee3e-11e7-9220-000c29bfdc52)" can be found. Need to start a new one
E1231 23:22:08.857121   15369 cni.go:259] Error adding network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
E1231 23:22:08.857144   15369 cni.go:227] Error while adding to cni network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
E1231 23:22:08.930343   15369 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "busybox_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory

多次實驗,導致node的網絡配置汙染,所以新驗證時一定要reboot,清理環境

rm -rf /var/lib/kubelet/
docker rm $(docker ps -a -q)

controller需要加這兩個參數,且cluster-cidr的地址和cni-flannel.yaml裏的地址要一致.和svc網段的地址(--service-cluster-ip-range)不要相同.

--allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16

no IP addresses available in range set: 10.244.0.1-10.244.0.254

沒地址了,pod一直在創建中....
參考: https://github.com/kubernetes/kubernetes/issues/57280

現象:

- kubelet報錯
E0101 00:06:38.629105    1109 kuberuntime_manager.go:647] createPodSandbox for pod "busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "busybox2_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254
E0101 00:06:38.629143    1109 pod_workers.go:186] Error syncing pod 7fa06467-ee44-11e7-a440-000c29bfdc52 ("busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)"), skipping: failed to "CreatePodSandbox" for "busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)" with CreatePodSandboxError: "CreatePodSandbox for pod \"busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"busybox2_default\" network: failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254"


- ip地址

/var/lib/cni/networks# ls cbr0/
10.244.0.10   10.244.0.123  10.244.0.147  10.244.0.170  10.244.0.194  10.244.0.217  10.244.0.240  10.244.0.35  10.244.0.59  10.244.0.82
10.244.0.100  10.244.0.124  10.244.0.148  10.244.0.171  10.244.0.195  10.244.0.218  10.244.0.241  10.244.0.36  10.244.0.6   10.244.0.83
10.244.0.101  10.244.0.125  10.244.0.149  10.244.0.172  10.244.0.196  10.244.0.219  10.244.0.242  10.244.0.37  10.244.0.60  10.244.0.84
10.244.0.102  10.244.0.126  10.244.0.15   10.244.0.173  10.244.0.197  10.244.0.22   10.244.0.243  10.244.0.38  10.244.0.61  10.244.0.85
10.244.0.103  10.244.0.127  10.244.0.150  10.244.0.174  10.244.0.198  10.244.0.220  10.244.0.244  10.244.0.39  10.244.0.62  10.244.0.86
10.244.0.104  10.244.0.128  10.244.0.151  10.244.0.175  10.244.0.199  10.244.0.221  10.244.0.245  10.244.0.4   10.244.0.63  10.244.0.87
10.244.0.105  10.244.0.129  10.244.0.152  10.244.0.176  10.244.0.2    10.244.0.222  10.244.0.246  10.244.0.40  10.244.0.64  10.244.0.88
10.244.0.106  10.244.0.13   10.244.0.153  10.244.0.177  10.244.0.20   10.244.0.223  10.244.0.247  10.244.0.41  10.244.0.65  10.244.0.89
10.244.0.107  10.244.0.130  10.244.0.154  10.244.0.178  10.244.0.200  10.244.0.224  10.244.0.248  10.244.0.42  10.244.0.66  10.244.0.9
10.244.0.108  10.244.0.131  10.244.0.155  10.244.0.179  10.244.0.201  10.244.0.225  10.244.0.249  10.244.0.43  10.244.0.67  10.244.0.90
10.244.0.109  10.244.0.132  10.244.0.156  10.244.0.18   10.244.0.202  10.244.0.226  10.244.0.25   10.244.0.44  10.244.0.68  10.244.0.91
10.244.0.11   10.244.0.133  10.244.0.157  10.244.0.180  10.244.0.203  10.244.0.227  10.244.0.250  10.244.0.45  10.244.0.69  10.244.0.92
10.244.0.110  10.244.0.134  10.244.0.158  10.244.0.181  10.244.0.204  10.244.0.228  10.244.0.251  10.244.0.46  10.244.0.7   10.244.0.93
10.244.0.111  10.244.0.135  10.244.0.159  10.244.0.182  10.244.0.205  10.244.0.229  10.244.0.252  10.244.0.47  10.244.0.70  10.244.0.94
10.244.0.112  10.244.0.136  10.244.0.16   10.244.0.183  10.244.0.206  10.244.0.23   10.244.0.253  10.244.0.48  10.244.0.71  10.244.0.95
10.244.0.113  10.244.0.137  10.244.0.160  10.244.0.184  10.244.0.207  10.244.0.230  10.244.0.254  10.244.0.49  10.244.0.72  10.244.0.96
10.244.0.114  10.244.0.138  10.244.0.161  10.244.0.185  10.244.0.208  10.244.0.231  10.244.0.26   10.244.0.5   10.244.0.73  10.244.0.97
10.244.0.115  10.244.0.139  10.244.0.162  10.244.0.186  10.244.0.209  10.244.0.232  10.244.0.27   10.244.0.50  10.244.0.74  10.244.0.98
10.244.0.116  10.244.0.14   10.244.0.163  10.244.0.187  10.244.0.21   10.244.0.233  10.244.0.28   10.244.0.51  10.244.0.75  10.244.0.99
10.244.0.117  10.244.0.140  10.244.0.164  10.244.0.188  10.244.0.210  10.244.0.234  10.244.0.29   10.244.0.52  10.244.0.76  last_reserved_ip.0
10.244.0.118  10.244.0.141  10.244.0.165  10.244.0.189  10.244.0.211  10.244.0.235  10.244.0.3    10.244.0.53  10.244.0.77
10.244.0.119  10.244.0.142  10.244.0.166  10.244.0.19   10.244.0.212  10.244.0.236  10.244.0.30   10.244.0.54  10.244.0.78
10.244.0.12   10.244.0.143  10.244.0.167  10.244.0.190  10.244.0.213  10.244.0.237  10.244.0.31   10.244.0.55  10.244.0.79
10.244.0.120  10.244.0.144  10.244.0.168  10.244.0.191  10.244.0.214  10.244.0.238  10.244.0.32   10.244.0.56  10.244.0.8
10.244.0.121  10.244.0.145  10.244.0.169  10.244.0.192  10.244.0.215  10.244.0.239  10.244.0.33   10.244.0.57  10.244.0.80
10.244.0.122  10.244.0.146  10.244.0.17   10.244.0.193  10.244.0.216  10.244.0.24   10.244.0.34   10.244.0.58  10.244.0.81

- flannel創建了很多文件
/var/lib/cni/flannel#  ls | wc ; date

解決:

幹掉cni-flannel,停運集群.清理環境.

rm -rf /var/lib/cni/flannel/* && rm -rf /var/lib/cni/networks/cbr0/* && ip link delete cni0
rm -rf  /var/lib/cni/networks/cni0/*
  • 推薦打開,不打開我沒發現什麽問題
    ```
    echo ‘net.bridge.bridge-nf-call-iptables=1‘ >> /etc/sysctl.conf
    sysctl -p

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1


參考:
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
http://cizixs.com/2017/05/23/container-network-cni
https://k8smeetup.github.io/docs/concepts/cluster-administration/network-plugins/
https://mritd.me/2017/09/20/set-up-ha-kubernetes-cluster-on-aliyun-ecs/
https://coreos.com/flannel/docs/latest/kubernetes.html  (不要用legency的那個,那個是kubelet模式)
https://feisky.gitbooks.io/kubernetes/network/flannel/#cni集成
http://blog.csdn.net/idea77/article/details/78793318




## kube-proxy ipvs模式

目前還是測試版,打開玩一玩.
參考: https://jicki.me/2017/12/20/kubernetes-1.9-ipvs/#%E5%90%AF%E5%8A%A8-kube-proxy
https://mritd.me/2017/10/10/kube-proxy-use-ipvs-on-kubernetes-1.8/

確保內核有rr模塊
[root@n2 ~]# lsmod | grep ip_vs
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 2
ip_vs 141092 8 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 133387 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack

啟用 ipvs 後與 1.7 版本的配置差異如下：

增加 --feature-gates=SupportIPVSProxyMode=true 選項，用於告訴 kube-proxy 開啟 ipvs 支持，因為目前 ipvs 並未穩定
增加 ipvs-min-sync-period、--ipvs-sync-period、--ipvs-scheduler 三個參數用於調整 ipvs，具體參數值請自行查閱 ipvs 文檔
增加 --masquerade-all 選項，以確保反向流量通過

重點說一下 --masquerade-all 選項: kube-proxy ipvs 是基於 NAT 實現的，當創建一個 service 後，kubernetes 會在每個節點上創建一個網卡，同時幫你將 Service IP(VIP) 綁定上，此時相當於每個 Node 都是一個 ds，而其他任何 Node 上的 Pod，甚至是宿主機服務(比如 kube-apiserver 的 6443)都可能成為 rs；按照正常的 lvs nat 模型，所有 rs 應該將 ds 設置成為默認網關，以便數據包在返回時能被 ds 正確修改；在 kubernetes 將 vip 設置到每個 Node 後，默認路由顯然不可行，所以要設置 --masquerade-all 選項，以便反向數據包能通過

註意:--masquerade-all 選項與 Calico 安全策略控制不兼容,請酌情使用

kube-proxy --master=http://192.168.14.11:8080 --v=2 --feature-gates=SupportIPVSProxyMode=true --masquerade-all --proxy-mode=ipvs --masquerade-all

註意:
1.需要打開 –feature-gates=SupportIPVSProxyMode=true,官方 –feature-gates=SupportIPVSProxyMode=false 默認是 false
2.–masquerade-all 必須添加這項配置，否則 創建 svc 在 ipvs 不會添加規則
3.打開 ipvs 需要安裝 ipvsadm 軟件， 在 node 中安裝
yum install ipvsadm -y
ipvsadm -L -n

[root@n2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr persistent 10800
-> 192.168.14.11:6443 Masq 1 0 0
TCP 10.254.12.188:80 rr
-> 10.244.0.3:80 Masq 1 0 0
-> 10.244.1.4:80 Masq 1 0 0
```

[k8s]k8s 1.9(on the fly搭建) 1.9_cni-flannel部署排錯 ipvs模式

相關推薦

[k8s]k8s 1.9(on the fly搭建) 1.9_cni-flannel部署排錯 ipvs模式

wap sch byte cer ica 同步 disable cred 0.12 角色 節點名 節點ip master n1 192.168.14.11 節點1 n2 192.168.14.12 節點2 n3 192.168.14.13 htt

[K8s 1.9實踐]Kubeadm 1.9 HA 高可用 叢集 本地離線映象部署_Kubernetes中文社群

Kubeadm HA 1.9 高可用 叢集 本地離線部署 k8s介紹 k8s 發展速度很快，目前很多大的公司容器叢集都基於該專案，如京東，騰訊，滴滴，瓜子二手車,北森等等。 kubernetes1.9版本釋出2017年12月15日，每是那三個月一個迭代， Workloads API成為穩定版本，這消除

Image resize on-the-fly with Lambda and S3

A crash course on Serverless with AWS — Image resize on-the-fly with Lambda and S3Handling large images has always been a pain in my side since I started w

Marginally Interesting: Command Line Interactive Machine Learning on the JVM. Part 1: Why?

Tweet Just to get this out of the way first, no, I’m not going to talk

React Native FlatList強制重新整理Changing numColumns on the fly is not supported

紅色的喜慶顏色又來了。 這個報錯很明白，就是不讓你重新整理，因為你的key沒變，我猜他重新整理的判斷key沒變所以判斷資料沒變無需重新整理，但是實際上資料確實變了，那我們就把key變一下把 _renderItem = (info) => {

ubuntu18.04 flink-1.9.0 Standalone叢集搭建

叢集規劃 Master JobManager Standby JobManager Task Manager Zookeeper flink01 √ √ flink02 √ √ flink03 √ √ √ flink04 √ √ √ 前置準備 克隆4臺虛擬機器

安裝SVN（TortoiseSVN-1.9.7.27907-x64-svn-1.9.7.msi） 的步驟及安裝過程中的遇到的2502，2503問題，和如何解決這種情況的問題

       就在今天，我突然發現我的SVN的圖示沒有了，之前明明有安裝過SVN，於是我就找之前安裝的軟體的資料夾，沒有找到，記得得我好想沒有解除安裝，沒有找到，我只有重灌了。結果重灌軟體好像不行啊，貌似想到了什麼就去控制面板上看下有沒有這個軟體，結果在控制面板上看到了還有

純手工搭建kubernetes(k8s)1.9集群 - (二)核心模塊部署

kubernetes 集群部署 環境搭建 devops 持續集成 1. 部署ETCD（主節點） 1.1 簡介 ??kubernetes需要存儲很多東西，像它本身的節點信息，組件信息，還有通過kubernetes運行的pod，deployment，service等等。都需要持久化。etcd就

k8s 1.9.0-手動安裝-2

none working ole pac value kill swap port sch 1 下載etcd新版 https://github.com/coreos/etcd/releases 直接下載k8s的二進制包 https://github.com/kubernet

k8s 1.9.1 centos7 install 離線安裝

article sudo -a blog .net 密碼 persist per star 參考 http://blog.csdn.net/acxlm/article/details/79069468遇到的問題及解決：1. 直接用aliyun docker 可以用最新的，

K8S 1.9.0二進制包部署（三）

無法 emc mod tps -o cal text tor amd 4、k8s node#############kubelet cp kubelet /usr/bin/chmod 755 /usr/bin/kubeletmkdir -p /app/kubernetes/

K8S 1.9.0二進制包部署（五）

alt ces map ima config -c tex sta ffffff 6、dashboard v1.8.0在kubernetes目錄下有src的壓縮包，解壓後在cluster/addons/dashboard 下有5個文件dashboard-controller

K8S 1.9.0二進制包部署（四）

ble forward term ecs core bus des usr ets 5、flannel cp flanneld /usr/bin/chmod 755 /usr/bin/flanneldmkdir -p /app/flannel/conf/ vi /usr/l

安裝k8s 1.9.0 實踐：問題集錦

[[email protected] ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/etc/systemd/system/kubelet.

k8s（一）、 1.9.0高可用叢集本地離線部署記錄

一、部署說明 1.節點 master1: IP:192.168.0.170/24 hostname：171 master2: IP:192.168.0.171/24 hostname：172 VIP:192.168.0.169/24 2.工具版

Testlink安裝：Notice:Undefined index: type in C:inetpubwwwroot estlink-1.9.3installinstallCheck.php on line 41

ndk coq vip cbt fhq mft ryu base64 gb2 問題現象： 問題原因：php配置參數中錯誤提示顯示； 問題解決：修改php.ini配置文件，修改為如下： error_reporting = E_ALL & ~E_NOTICE

jQuery.1.9 live 代替事件 on 新增內容無法觸發事件

class status jquer earch pen cnblogs span == nbsp 如果是新增 append 或者 html() 事件添加的內容,無法觸發 click 事件, 在1.9 可以用live 事件來代替 1.9以後用 <div class

[ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-apiserver-amd64:v1.11.1]: exit status 1

occurred cidr .com src cal 4.0 schedule -a uber 問題描述 [root@localhost ~]# kubeadm init --kubernetes-version=v1.11.1 --pod-network-cidr=10.

k8s升級，HA叢集1.12.0～HA叢集1.13.0

k8s升級，此次升級是1.12.0 至1.13.0 準備 # 首先升級master節點的基礎元件kubeadm、kubelet、kubectl apt-get update && apt-get upgrade kubeadm # 更新到1.13.0後，在master執行,執行一次

K8S運維筆記-單節點叢集安裝(1.11版本)

使用kubeadm安裝K8S 安裝kubeadm 更新安裝源 cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] nam