kubernetes匯出有狀態服務
kubernetes匯出有狀態服務
(金慶的專欄 2018.7)
網遊伺服器中的房間伺服器是有狀態伺服器,可以用 kubernetes statefulset 開啟多個例項。
為了讓客戶端能夠直連房間伺服器,除了 statefulset 要求的 headless 服務,
還須為每個例項建立 NodePort 型別的服務, 並且選擇Pod和禁止轉發。
下面 bootcamp.yml 先建立了 bootcamp headless 服務(clusterIP: None),
又建立了 bootcamp StatefulSet, 例項個數為 2.
然後建立 bootcamp-0,1,2 服務,分別對應 bootcamp-0,1,2 pod.
服務個數大於例項個數,是想測試下服務沒有對應的例項時的表現。
網遊中的匹配伺服器將分配一個房間給客戶端,列舉 bootcamp-0,1,2 pod 所在節點的外網 IP,
連同對應服務的埠,傳送給客戶端,讓客戶端直連。
[[email protected]10-240-79-10 statefulset]$ cat bootcamp.yml
apiVersion: v1
kind: Service
metadata:
name: bootcamp
namespace: jq
labels:
name: bootcamp
spec:
ports:
- port: 8080
clusterIP: None # StatefulSet要求Headless服務
selector:
app: bootcamp # 選擇 bootcamp 應用
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: bootcamp
namespace: jq
spec:
serviceName: bootcamp # 上面的 Headless 服務名
replicas: 2
template:
metadata:
labels:
app: bootcamp # 應用名,與服務中的 selector 對應
spec:
containers:
- name: bootcamp
image: docker.io/jocatalin/kubernetes-bootcamp:v1
---
kind: Service
apiVersion: v1
metadata:
name: bootcamp-0
namespace: jq
spec:
type: NodePort # 對外服務
externalTrafficPolicy: Local # 不要轉發到其他節點
selector:
app: bootcamp
statefulset.kubernetes.io/pod-name: bootcamp-0 # 選擇 pod
ports:
- protocol: TCP
nodePort: 30880 # 對外埠
port: 8080
---
kind: Service
apiVersion: v1
metadata:
name: bootcamp-1
namespace: jq
spec:
type: NodePort
externalTrafficPolicy: Local
selector:
app: bootcamp
statefulset.kubernetes.io/pod-name: bootcamp-1
ports:
- protocol: TCP
nodePort: 30881
port: 8080
---
kind: Service
apiVersion: v1
metadata:
name: bootcamp-2
namespace: jq
spec:
type: NodePort
externalTrafficPolicy: Local
selector:
app: bootcamp
statefulset.kubernetes.io/pod-name: bootcamp-2
ports:
- protocol: TCP
nodePort: 30882
port: 8080
因為 statefulset 的每個例項有不同的標籤,所以可以為服務選擇一個例項。
利用 externalTrafficPolicy: Local 設定來禁止轉發。
參考 service.spec.externalTrafficPolicy 的說明:
https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport
Setting service.spec.externalTrafficPolicy to the value Local will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. If there are no local endpoints, packets sent to the node are dropped, …
因為有可能多個Pod開在同一節點上,所以對外埠設成了不同的 30880-30882。
如果限制每個節點只開一個例項,則對外埠可以設成同一個。
建立服務:
[jinqing@host-10-240-79-10 statefulset]$ kubectl apply -f bootcamp.yml
service "bootcamp" created
statefulset.apps "bootcamp" created
service "bootcamp-0" created
service "bootcamp-1" created
service "bootcamp-2" created
服務如下:
[[email protected] statefulset]$ kubectl get service -n jq
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bootcamp ClusterIP None <none> 8080/TCP 3m
bootcamp-0 NodePort 10.96.128.137 <none> 8080:30880/TCP 3m
bootcamp-1 NodePort 10.109.2.56 <none> 8080:30881/TCP 3m
bootcamp-2 NodePort 10.102.181.193 <none> 8080:30882/TCP 3m
2個例項:
[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME READY STATUS RESTARTS AGE IP NODE
bootcamp-0 1/1 Running 0 4m 10.244.0.42 host-10-240-79-10
bootcamp-1 1/1 Running 0 4m 10.244.1.63 host-10-240-79-11
訪問服務必須指定節點,不會自動轉發:
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30880
Hello Kubernetes bootcamp! | Running on: bootcamp-0 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30881
curl: (7) Failed connect to 10.240.79.10:30881; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30880
curl: (7) Failed connect to 10.240.79.11:30880; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
沒有負載均衡,30881埠總是訪問 bootcamp-1:
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
也可以從外網訪問.
30882 埠無法連線:
[[email protected] statefulset]$ curl 10.240.79.11:30882
curl: (7) Failed connect to 10.240.79.11:30882; Connection refused
[[email protected] statefulset]$ curl 10.240.79.10:30882
curl: (7) Failed connect to 10.240.79.10:30882; Connection refused
3個埠都有監聽:
[root@host-10-240-79-11 tmp]# netstat -ntl | grep 3088
tcp6 0 0 :::30881 :::* LISTEN
tcp6 0 0 :::30882 :::* LISTEN
tcp6 0 0 :::30880 :::* LISTEN
iptables-save 輸出如下, 其中 10.244是Pod的網段。
沒有例項執行的節點上,會丟棄請求:
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH
-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "jq/bootcamp-1: has no local endpoints" -j KUBE-MARK-DROP
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP
有例項執行的節點上會轉發給 Pod 8080:
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "Balancing rule 0 for jq/bootcamp-1:" -j KUBE-SEP-LJQA4WUIKJUQ5ALU
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP
-A KUBE-SEP-LJQA4WUIKJUQ5ALU -s 10.244.1.63/32 -m comment --comment "jq/bootcamp-1:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LJQA4WUIKJUQ5ALU -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp -j DNAT --to-destination 10.244.1.63:8080
30882 埠無法連線
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "jq/bootcamp-2: has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 30882 -j REJECT --reject-with icmp-port-unreachable
測試下擴容:
[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME DESIRED CURRENT AGE
bootcamp 2 2 45m
[jinqing@host-10-240-79-10 statefulset]$ kubectl scale --replicas=3 statefulset/bootcamp -n jq
statefulset.apps "bootcamp" scaled
[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME DESIRED CURRENT AGE
bootcamp 3 3 47m
[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME READY STATUS RESTARTS AGE IP NODE
bootcamp-0 1/1 Running 0 48m 10.244.0.42 host-10-240-79-10
bootcamp-1 1/1 Running 0 48m 10.244.1.63 host-10-240-79-11
bootcamp-2 1/1 Running 0 45s 10.244.2.60 host-10-240-79-12
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.12:30882
Hello Kubernetes bootcamp! | Running on: bootcamp-2 | v=1