第六章 網絡編程-SOCKET開發——續2
6.5——粘包現象與解決方案
簡單遠程執行命令程序開發
是時候用戶socket幹點正事呀,我們來寫一個遠程執行命令的程序,寫一個socket client端在windows端發送指令,一個socket server在Linux端執行命令並返回結果給客戶端
執行命令的話,肯定是用我們學過的subprocess模塊啦,但註意註意註意:
res=subprocess.Popen(cmd.deocde(‘utf-8‘),shell=subprocess.PIPE,stdout=subprocess.PIPE
命令結果的編碼是以當前所在的系統為準的,如果是windows,那麽res.stdout.read()讀出的就是GBK編碼的
ssh server
import socket import subprocess ip_port = (‘127.0.0.1‘, 8080) tcp_socket_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) tcp_socket_server.bind(ip_port) tcp_socket_server.listen(5) while True: conn, addr = tcp_socket_server.accept() print(‘客戶端‘, addr) while True: cmd = conn.recv(1024) if len(cmd) == 0: break print("recv cmd",cmd) res = subprocess.Popen(cmd.decode(‘utf-8‘), shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) stderr = res.stderr.read() stdout = res.stdout.read() print("res length",len(stdout)) conn.send(stderr) conn.send(stdout)
ssh client
import socket ip_port = (‘127.0.0.1‘, 8080) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) res = s.connect_ex(ip_port) while True: msg = input(‘>>: ‘).strip() if len(msg) == 0: continue if msg == ‘quit‘: break s.send(msg.encode(‘utf-8‘)) act_res = s.recv(1024) print(act_res.decode(‘utf-8‘), end=‘‘)
粘包的解決辦法
問題的根源在於,接收端不知道發送端將要傳送的字節流的長度,所以解決粘包的方法就是圍繞,如何讓發送端在發送數據前,把自己將要發送的字節流總大小讓接收端知曉,然後接收端來一個死循環接收完所有數據
普通版
服務器端
import socket,subprocess ip_port=(‘127.0.0.1‘,8080) s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(ip_port) s.listen(5) while True: conn,addr=s.accept() print(‘客戶端‘,addr) while True: msg=conn.recv(1024) if not msg:break res=subprocess.Popen(msg.decode(‘utf-8‘),shell=True, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE) err=res.stderr.read() if err: ret=err else: ret=res.stdout.read() data_length=len(ret) conn.send(str(data_length).encode(‘utf-8‘)) data=conn.recv(1024).decode(‘utf-8‘) if data == ‘recv_ready‘: conn.sendall(ret) conn.close()
客戶端
import socket,time s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) res=s.connect_ex((‘127.0.0.1‘,8080)) while True: msg=input(‘>>: ‘).strip() if len(msg) == 0:continue if msg == ‘quit‘:break s.send(msg.encode(‘utf-8‘)) length=int(s.recv(1024).decode(‘utf-8‘)) s.send(‘recv_ready‘.encode(‘utf-8‘)) send_size=0 recv_size=0 data=b‘‘ while recv_size < length: data+=s.recv(1024) recv_size+=len(data) #為什麽不直接寫1024? print(data.decode(‘utf-8‘))
為何low?
程序的運行速度遠快於網絡傳輸速度,所以在發送一段字節前,先用send去發送該字節流長度,這種方式會放大網絡延遲帶來的性能損耗
剛才上面 在發送消息之前需先發送消息長度給對端,還必須要等對端返回一個ready收消息的確認,不等到對端確認就直接發消息的話,還是會產生粘包問題(承載消息長度的那條消息和消息本身粘在一起)。 有沒有優化的好辦法麽?
文藝版
思考一個問題,為什麽不能在發送了消息長度(稱為消息頭head吧)給對端後,立刻發消息內容(稱為body吧),是因為怕head 和body 粘在一起,所以通過等對端返回確認來把兩條消息中斷開。
可不可以直接發head + body,但又能讓對端區分出哪個是head,哪個是body呢?我靠、我靠,感覺智商要漲了。
想到了,把head設置成定長的呀,這樣對端只要收消息時,先固定收定長的數據,head裏寫好,後面還有多少是屬於這條消息的數據,然後直接寫個循環收下來不就完了嘛!唉呀媽呀,我真機智。
可是、可是如何制作定長的消息頭呢?假設你有2條消息要發送,第一條消息長度是 3000個字節,第2條消息是200字節。如果消息頭只包含消息長度的話,那兩個消息的消息頭分別是
len(msf1)=4000=4字節
len(msg2)=200=3字節
你的服務端如何完整的收到這個消息頭呢?是recv(3)還是recv(4)服務器端怎麽知道?用盡我所有知識,我只能想到拼接字符串的辦法了,打比方就是設置消息頭固定100字節長,不夠的拿空字符串去拼接。
server
import socket,json import subprocess ip_port = (‘127.0.0.1‘, 8080) tcp_socket_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) tcp_socket_server.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1) #一行代碼搞定,寫在bind之前 tcp_socket_server.bind(ip_port) tcp_socket_server.listen(5) def pack_msg_header(header,size): bytes_header = bytes(json.dumps(header),encoding="utf-8") fill_up_size = size - len(bytes_header) print("need to fill up ",fill_up_size) header[‘fill‘] = header[‘fill‘].zfill(fill_up_size) print("new header",header) bytes_new_header = bytes(bytes(json.dumps(header),encoding="utf-8")) return bytes_new_header while True: conn, addr = tcp_socket_server.accept() print(‘客戶端‘, addr) while True: cmd = conn.recv(1024) if len(cmd) == 0: break print("recv cmd",cmd) res = subprocess.Popen(cmd.decode(‘utf-8‘), shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) stderr = res.stderr.read() stdout = res.stdout.read() print("res length",len(stdout)) msg_header = { ‘length‘:len(stdout + stderr), ‘fill‘:‘‘ } packed_header = pack_msg_header(msg_header,100) print("packed header size",packed_header,len(packed_header)) conn.send(packed_header) conn.send(stdout + stderr)
client
import socket import json ip_port = (‘127.0.0.1‘, 8080) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) res = s.connect_ex(ip_port) while True: msg = input(‘>>: ‘).strip() if len(msg) == 0: continue if msg == ‘quit‘: break s.send(msg.encode(‘utf-8‘)) response_msg_header = s.recv(100).decode("utf-8") response_msg_header_data = json.loads(response_msg_header) msg_size = response_msg_header_data[‘length‘] res = s.recv(msg_size) print("received res size ",len(res)) print(res.decode(‘utf-8‘), end=‘‘)
文藝版二
為字節流加上自定義固定長度報頭也可以借助第三方模塊struct,用法為
import json,struct #假設通過客戶端上傳1T:1073741824000的文件a.txt #為避免粘包,必須自定制報頭 header={‘file_size‘:1073741824000,‘file_name‘:‘/a/b/c/d/e/a.txt‘,‘md5‘:‘8f6fbf8347faa4924a76856701edb0f3‘} #1T數據,文件路徑和md5值 #為了該報頭能傳送,需要序列化並且轉為bytes head_bytes=bytes(json.dumps(header),encoding=‘utf-8‘) #序列化並轉成bytes,用於傳輸 #為了讓客戶端知道報頭的長度,用struck將報頭長度這個數字轉成固定長度:4個字節 head_len_bytes=struct.pack(‘i‘,len(head_bytes)) #這4個字節裏只包含了一個數字,該數字是報頭的長度 #客戶端開始發送 conn.send(head_len_bytes) #先發報頭的長度,4個bytes conn.send(head_bytes) #再發報頭的字節格式 conn.sendall(文件內容) #然後發真實內容的字節格式 #服務端開始接收 head_len_bytes=s.recv(4) #先收報頭4個bytes,得到報頭長度的字節格式 x=struct.unpack(‘i‘,head_len_bytes)[0] #提取報頭的長度 head_bytes=s.recv(x) #按照報頭長度x,收取報頭的bytes格式 header=json.loads(json.dumps(header)) #提取報頭 #最後根據報頭的內容提取真實的數據,比如 real_data_len=s.recv(header[‘file_size‘]) s.recv(real_data_len)
使用struct模塊實現方式如下
server
import socket,struct,json import subprocess phone=socket.socket(socket.AF_INET,socket.SOCK_STREAM) phone.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1) #就是它,在bind前加 phone.bind((‘127.0.0.1‘,8080)) phone.listen(5) while True: conn,addr=phone.accept() while True: cmd=conn.recv(1024) if not cmd:break print(‘cmd: %s‘ %cmd) res=subprocess.Popen(cmd.decode(‘utf-8‘), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) err=res.stderr.read() print(err) if err: back_msg=err else: back_msg=res.stdout.read() headers={‘data_size‘:len(back_msg)} head_json=json.dumps(headers) head_json_bytes=bytes(head_json,encoding=‘utf-8‘) conn.send(struct.pack(‘i‘,len(head_json_bytes))) #先發報頭的長度 conn.send(head_json_bytes) #再發報頭 conn.sendall(back_msg) #在發真實的內容 conn.close()
client
from socket import * import struct,json ip_port=(‘127.0.0.1‘,8080) client=socket(AF_INET,SOCK_STREAM) client.connect(ip_port) while True: cmd=input(‘>>: ‘) if not cmd:continue client.send(bytes(cmd,encoding=‘utf-8‘)) head=client.recv(4) #先收4個bytes,這裏4個bytes裏包含了報頭的長度 head_json_len=struct.unpack(‘i‘,head)[0] #解出報頭的長度 head_json=json.loads(client.recv(head_json_len).decode(‘utf-8‘)) #拿到報頭 data_len=head_json[‘data_size‘] #取出報頭內包含的信息 #開始收數據 recv_size=0 recv_data=b‘‘ while recv_size < data_len: recv_data+=client.recv(1024) recv_size+=len(recv_data) print(recv_data.decode(‘utf-8‘))
6.6——通過socket發送文件
通過socket收發文件軟件開發
1、客戶端提交命令 2、服務端接收命令,解析,執行下載文件的方法,即以讀的方式打開文件,for循環讀出文件的一行行內容,然後send給客戶端 3、客戶端以寫的方式打開文件,將接收的內容寫入文件中
參照上一小節文藝青年實現版二,示範代碼如下
服務端實現
import socket import struct import json import subprocess import os class MYTCPServer: address_family = socket.AF_INET socket_type = socket.SOCK_STREAM allow_reuse_address = False max_packet_size = 8192 coding=‘utf-8‘ request_queue_size = 5 server_dir=‘file_upload‘ def __init__(self, server_address, bind_and_activate=True): """Constructor. May be extended, do not override.""" self.server_address=server_address self.socket = socket.socket(self.address_family, self.socket_type) if bind_and_activate: try: self.server_bind() self.server_activate() except: self.server_close() raise def server_bind(self): """Called by constructor to bind the socket. """ if self.allow_reuse_address: self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.socket.bind(self.server_address) self.server_address = self.socket.getsockname() def server_activate(self): """Called by constructor to activate the server. """ self.socket.listen(self.request_queue_size) def server_close(self): """Called to clean-up the server. """ self.socket.close() def get_request(self): """Get the request and client address from the socket. """ return self.socket.accept() def close_request(self, request): """Called to clean up an individual request.""" request.close() def run(self): while True: self.conn,self.client_addr=self.get_request() print(‘from client ‘,self.client_addr) while True: try: head_struct = self.conn.recv(4) if not head_struct:break head_len = struct.unpack(‘i‘, head_struct)[0] head_json = self.conn.recv(head_len).decode(self.coding) head_dic = json.loads(head_json) print(head_dic) #head_dic={‘cmd‘:‘put‘,‘filename‘:‘a.txt‘,‘filesize‘:123123} cmd=head_dic[‘cmd‘] if hasattr(self,cmd): func=getattr(self,cmd) func(head_dic) except Exception: break def put(self,args): file_path=os.path.normpath(os.path.join( self.server_dir, args[‘filename‘] )) filesize=args[‘filesize‘] recv_size=0 print(‘----->‘,file_path) with open(file_path,‘wb‘) as f: while recv_size < filesize: recv_data=self.conn.recv(self.max_packet_size) f.write(recv_data) recv_size+=len(recv_data) print(‘recvsize:%s filesize:%s‘ %(recv_size,filesize)) tcpserver1=MYTCPServer((‘127.0.0.1‘,8080)) tcpserver1.run()
客戶端
import socket import struct import json import os class MYTCPClient: address_family = socket.AF_INET socket_type = socket.SOCK_STREAM allow_reuse_address = False max_packet_size = 8192 coding=‘utf-8‘ request_queue_size = 5 def __init__(self, server_address, connect=True): self.server_address=server_address self.socket = socket.socket(self.address_family, self.socket_type) if connect: try: self.client_connect() except: self.client_close() raise def client_connect(self): self.socket.connect(self.server_address) def client_close(self): self.socket.close() def run(self): while True: inp=input(">>: ").strip() if not inp:continue l=inp.split() cmd=l[0] if hasattr(self,cmd): func=getattr(self,cmd) func(l) def put(self,args): cmd=args[0] filename=args[1] if not os.path.isfile(filename): print(‘file:%s is not exists‘ %filename) return else: filesize=os.path.getsize(filename) head_dic={‘cmd‘:cmd,‘filename‘:os.path.basename(filename),‘filesize‘:filesize} print(head_dic) head_json=json.dumps(head_dic) head_json_bytes=bytes(head_json,encoding=self.coding) head_struct=struct.pack(‘i‘,len(head_json_bytes)) self.socket.send(head_struct) self.socket.send(head_json_bytes) send_size=0 with open(filename,‘rb‘) as f: for line in f: self.socket.send(line) send_size+=len(line) print(send_size) else: print(‘upload successful‘) client=MYTCPClient((‘127.0.0.1‘,8080)) client.run()
第六章 網絡編程-SOCKET開發——續2