【原】k8s ingress-nginx 針對指定 User-Agent 爬蟲進行限速
阿新 • • 發佈:2020-12-12
針對指定 User-Agent 進行限速
修改模板
將 ingress-nginx 模板nginx.tmpl
提取出來,設定成 configmap 掛載到 ingress-nginx 中去使用
新增 map
新增如下內容到模板中
533 map $http_user_agent $agent { 534 default ""; 535 #~curl $http_user_agent; 536 ~*apachebench $http_user_agent; 537 ~*spider $http_user_agent; 538 ~*bot $http_user_agent; 539 ~*slurp $http_user_agent; 540 ~*hello $http_user_agent; 541 ~*chrome $http_user_agent; 542 } 543 544 limit_conn_zone $agent zone=conn_lyj_com:10m; 545 limit_req_zone $agent zone=req_lyj_com:10m rate=1r/s; 546 547 limit_req zone=req_lyj_com burst=1 nodelay;
第一種:返回錯誤頁面
新增錯誤頁面
針對指定 UA 新增錯誤頁面
新增 598-603 行內容到模板中
594 ## start server {{ $server.Hostname }} 595 server { 596 server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }}; 597 598 recursive_error_pages on; 599 proxy_intercept_errors on; 600 if ($http_user_agent !~ "(hello|chrome)"){ 601 set $err_page "https://www.qq.com/404/"; 602 } 603 error_page 503 $err_page; 604 605 {{ if gt (len $cfg.BlockUserAgents) 0 }} 606 if ($block_ua) { 607 return 403; 608 } 609 {{ end }} 610 {{ if gt (len $cfg.BlockReferers) 0 }} 611 if ($block_ref) { 612 return 403; 613 } 614 {{ end }} 615 616 {{ template "SERVER" serverConfig $all $server }} 617 618 {{ if not (empty $cfg.ServerSnippet) }} 619 # Custom code snippet configured in the configuration configmap 620 {{ $cfg.ServerSnippet }} 621 {{ end }}
實際檔案
容器中的檔案內容如下
481 ## start server lyj.5179.top 482 server { 483 server_name lyj.5179.top ; 484 485 recursive_error_pages on; 486 proxy_intercept_errors on; 487 if ($http_user_agent !~ "(hello|chrome)"){ 488 set $err_page "https://www.qq.com/404/"; 489 } 490 error_page 503 $err_page; 491 492 listen 80 ; 493 listen 443 ssl http2 ;
測試
指定 UA 為 hello
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
302
302
302
302
302
302
302
302
302
302
302
報錯日誌
10.32.0.1 - - [11/Dec/2020:09:24:15 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36" 486 0.000 [default-nginx-80] [] - - - - 621293a69b12f4b2f552ddefcfe677d5
2020/12/11 09:24:15 [error] 123#123: *1597 limiting requests, excess: 1.332 by zone "req_lyj_com", client: 10.32.0.1, server: lyj.5179.top, request: "GET / HTTP/1.1", host: "lyj.5179.top:30080"
不指定 UA
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -o /dev/null ;echo;done
200
200
200
200
200
200
200
200
200
第二種:不返回錯誤頁面,返回 444
新增錯誤碼
新增 598 行
594 ## start server {{ $server.Hostname }}
595 server {
596 server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
597
598 limit_req_status 444;
599
600 {{ if gt (len $cfg.BlockUserAgents) 0 }}
601 if ($block_ua) {
602 return 403;
603 }
604 {{ end }}
605 {{ if gt (len $cfg.BlockReferers) 0 }}
606 if ($block_ref) {
607 return 403;
608 }
測試
指定 ua
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
000
000
000
000
000
000
...
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -A"hello" ;echo;done
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
如果指定其他的狀態碼就不是 000 了
報連線被重置
日誌:
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - 5e431fb99cf77d32bc39d3657f6fc3ca
2020/12/11 12:01:03 [error] 38#38: *5185 limiting requests, excess: 1.896 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f58bc7feaa516f39cc9a4103fdb87ca5
2020/12/11 12:01:03 [error] 38#38: *5186 limiting requests, excess: 1.884 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f4d43616589ab93e4ae28c9815dd1d33
2020/12/11 12:01:03 [error] 37#37: *5187 limiting requests, excess: 1.872 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - b68eaaa44f8ebdcaa406c651b8b0f2ad
不指定 UA
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -o /dev/null ;echo;done
200
200
200
200
200
200
200
200