1. 程式人生 > 其它 >(轉) web日誌分析指令碼

(轉) web日誌分析指令碼

(1)日誌較大時,可選用如下命令對日誌分割
split -C 500m -a 2 src.log dst.log #按大小500M分割

(2)掃描器探測行為檢測
grep -E -i "AppScan|CustomCookie|netsparker|sqlmap|Havij|Pangolin|nessus|Openvas|whatWEB|w3af|DirBuster|WEBbench" xx.log >scan.txt

(3)攻擊行為檢測
grep -E -i "%27|%3c%3e|and|union|exist|select|version|update|script|alert|XSS|document|asa|\.\.|uploadfile|.jpg.[asp|aspx|jsp|php]|passwd|boot.ini|htaccess|WEB.xml|bak|svn|inc|config|conf|conn.asp|echo|mdb|cgi|dir|ipconfig|OPTIONS|PUT|HEAD|CMD|shell|info|bin|\(\)|cmd" -c xx.log

(4)木馬特徵檢測
grep -E -i "rootkit|3est|door|server|kim|phpspy|jspspy|command|shell|hack|f4ck|eval\(|system\(|.jpg.|editor|edit|fck" xx.log -c #木馬及編輯器檢測

(5)CC攻擊異常行為檢測
cat localhost_access_2014-09-12.log |awk '{print $1}'|sort |uniq -c|sort -nr|head -n 10

(6)訪問最多的頁面
cat localhost_access_2014-09-12.log |awk '{print $7}'|sort |uniq -c|sort -rn|head
cat localhost_access_2014-09-12.log |sed 's/^.*com/(.*/)/"//1/g'|awk '{print $7}'|sort |uniq -c|sort -rn|head #將域名內容去掉
cat access_log | grep "19/May/2010:00" | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 10
#當天訪問頁面排前10的url
cat access_log |cut -d ' ' -f 1 |sort |uniq -c | sort -nr | awk '{print $1 }' | head -n 10 |less #檢視日誌中訪問次數最多的前10個IP
cat access_log |cut -d ' ' -f 1 |sort |uniq -c | awk '{if ($1 > 100) print $0}'|sort -nr |less #檢視日誌中出現100次以上的IP
cat access_log |tail -10000|awk '{print $7}'|sort|uniq -c|sort -nr|less #檢視最近訪問量最高的檔案

(7)Windows遠端桌面異常登入
LogParser file:C:\temp\sec.evtx -o:chart -chartType:Bar3d -chartTitle:"TOP 10 URL"
Logparser -i:evt -o:csv "select * from C:\temp\sec.evtx where Message like '%登入型別: 10%' and EventID = 4624" > c:\temp\sec_log.csv
Logparser -i:evt -o:csv "select * from C:\temp\sec.evtx where Message like '%登入型別: 10%' and Message like '%已成功登入帳戶%' and Message not like '%192.168.%' and EventID = 4624" > c:\temp\sec_log_Type10_LoginSuccess_192.168.csv
Logparser -i:evt -o:csv "select * from C:\temp\sec.evtx where Message like '%已成功%' and Message not like '%192.168.%' and Message not like '%127.0.0.1%'" > c:\temp\sec_log_LoginSuccess_no192.168_no127.0.0.1.csv

(8)攻擊行為檢測
grep -E -i "test|shell|robots|backdoor|ma|mysql|sniffer|shacke|hack|diy|dbapp|fileupload|getpass|svchost|vnc|WEBproxy|root|mssql|help|sb|sql|cmd|rootkit|3est|door|server|kim|phpspy|jspspy|command|shell|f4ck|eval\(|system\(|edit|fck|manage|admin|houtai|guanli|super|denglu|login|AppScan|wvs|acunetix|CustomCookie|netsparker|sqlmap|Havij|Pangolin|nessus|Openvas|whatWEB|w3af|DirBuster|WEBbench|%27|%3c%3e|or|and|union|exists|select|version|update|order%20by|script|alert|XSS|onerror|msgbox|%3c%2f|prompt|document|\.asa|\.\.|uploadfile|\.jpg\.|passwd|\.ini|htaccess|WEB.xml|bak|svn|inc|conf|conn|echo|mdb|cgi|pl|\.sh|dir|ipconfig|OPTIONS|PUT|HEAD|CMD|info|bin|\(\)|inculde|systme|eval" xx.log >scan.txt

(9)篩選asp|jsp|php|shtml等常見格式
grep -E -i "\.asp|\.jsp|\.php|\.shtml|\.html|\.htm|\.ashx\.cgi|\.perl|\.xml|\.shtm|\.sh|\.nsp|\.do|\.action|\.ini|\.jpg\.|passwd|\.bak|\.svn|\.inc|\.conf|\.mdb|OPTIONS|PUT|HEAD|echo|access|\.asa|sql|shell|\.\.|php3|\.cfc|Servlet" 0401-0512menhu_edall.log>0401-0512menhu_edall_wz.log

(10)進一步篩選
針對以上2步篩選後仍然有大量資料的,進行3次篩選;根據第二步結果找出誤報較多的關鍵詞,並將其剔除。
grep -E -i "test|shell|backdoor|muma|mysql|sniffer|shacke|hack|diy|dbapp|fileupload|getpass|svchost|vnc|WEBproxy|root|mssql|help|sb|sql|cmd|rootkit|3est|door|server|kim|phpspy|jspspy|command|f4ck|eval\(|system\(|editor|fck|manage|admin|houtai|guanli|super|denglu|login|AppScan|wvs|acunetix|CustomCookie|netsparker|sqlmap|Havij|Pangolin|nessus|Openvas|whatWEB|w3af|DirBuster|WEBbench|%27|%3c%3e|or|and|union|exists|select|update|order%20by|script|alert|XSS|onerror|msgbox|%3c%2f|document|\.asa|\.\.|uploadfile|\.jpg\.|passwd|\.ini|htaccess|WEB.xml|bak|svn|inc|conf|echo|mdb|cgi|\.pl|\.sh|ipconfig|OPTIONS|PUT|HEAD|CMD|info|\(\)|inculde|systme|eval"

(11)定義Struts2遠端命令執行漏洞特徵
attackRule=denyMethodExecution|allowStaticMethodAccess

(12)定義SQL注入攻擊特徵
attackRule=(\w+)'|(\w+)%20and%20(\S+)|(\w+)%20or%20(\S+)|(\w+)=(\d+)-(\d+)|(\d+)>(\d+)|(\d+)<(\d+)|(\S)waitfor(\W+)delay(\S)|(\S)having(\W)|(\S)sleep(\W)|(\w)\+(\w)|(\w)\#|(\w)--|(\w)\/\*(\S)|(\w)\&\&(\W)|(\S)select(\W)|(\S)insert(\S+)into(\W)|(\S)delete(\W)|(\S)update(\W)|(\S)create(\W)|(\S)drop(\W)|(\S)exists(\W)|(\S)backup(\W)|(\S)order(\S+)by(\W)|(\S)group(\S+)by(\W)|(\S)exec(\S)|(\S)truncate(\S)|(\S)declare(\S)|(\S)@@version(\S)

(13)定義XSS攻擊特徵
attackRule=(\S)%3C(\S+)%3E|(\S)%3C(\S+)%2F%3E|(\S+)<(\S+)>|(\S+)<(\S+)\/>|onerror|onmouse|expression|\"|alert|document\.|prompt\(

(14)定義檔案包含和路徑遍歷攻擊特徵
attackRule=/etc/passwd|\/%c0%ae%c0%ae|\/%2E%2E|boot\.ini|win\.ini|\.\.\/|access\.log|httpd\.conf|nginx\.conf|/proc/self/environ

(15)定義常見WEBShell特徵
attackRule=\/cmd\.asp|\/diy\.asp|\.asp;|\/(\w+)\.(\w+)\/(\w+)\.php|\.php\.|eval\(|%eval|\.jsp?action=|fsaction=

(16)網站敏感檔案訪問
attackRule=\/WEB-INF\/WEB\.xml|applicationContext\.xml|\/manager\/html|\/jmx-console\/|\.properties|\.class|phpinfo\.php|\/conn\.asp|\/conn\.php|\/conn\.jsp

(17)木馬WEBshell及非法登陸檢測
grep -E -i "rootkit\.|3est\.|door\.|kim\.|phpspy\.|jspspy\.|\/command\.|shell\.|hack\.|f4ck\.|eval\(|system\(|\.jpg\.|action\.do|login|manager|super|editor|\/proc\/self\/environ|\/cmd\.|\/diy\.|\.asp;|ma\.|\/(\w+)\.(\w+)\/(\w+)\.php|\.php\.|eval\(|\%eval|\.jsp?action=|fsaction=|\/manage\/html|\/jmx-console\/|\.properties\.class|\/phpinfo\.|\/conn\.|\/config\." xx.log |awk '{print $7 $9}'|sort | uniq -c |sort -nr | grep -E "200$" |more

(18)所有攻擊行為檢測
grep -E -i "rootkit|3est|door|server|kim|phpspy|jspspy|command|shell|hack|f4ck|eval\(|system\(|\.jpg\.|AppScan|CustomCookie|netsparker|sqlmap|Havij|Pangolin|nessus|Openvas|whatWEB|w3af|DirBuster|WEBbench|OPTIONS|PUT|HEAD|DEBUG|system|action|login|manager|super|editor|(\w+)%27|(\w+)%20and%20(\S+)|(\w+)%20or%20(\S+)|(\w+)=(\d+)-(\d+)|(\d+)>(\d+)|(\d+)<(\d+)|(\S)waitfor(\W+)delay(\S)|(\S)having(\W)|(\S)sleep(\W)|(\w)\#|(\w)--|(\w)\/\*(\S)|(\w)\&\&(\W)|(\S)select(\W)|(\S)insert(\S+)into(\W)|(\S)delete(\W)|(\S)update(\W)|(\S)create(\W)|(\S)drop(\W)|(\S)exists(\W)|(\S)backup(\W)|(\S)order(\S+)by(\W)|(\S)group(\S+)by(\W)|(\S)exec(\S)|(\S)truncate(\S)|(\S)declare(\S)|(\S)@@version(\S)|(\S)%3C(\S+)%3E|(\S)%3C(\S+)%2F%3E|(\S+)<(\S+)>|(\S+)<(\S+)\/>|onerror|onmouse|expression|alert|document\.|prompt\(|\/etc\/passwd|\/%c0%ae%c0%ae|\/%2E%2E|boot\.ini|win\.ini|\.\.\/|access\.log|httpd\.conf|nginx\.conf|\/proc\/self\/environ|\/cmd\.asp|\/diy\.asp|\.asp;|\/(\w+)\.(\w+)\/(\w+)\.php|\.php\.|eval\(|%eval|\.jsp?action=|fsaction=|\/WEB-INF\/WEB\.xml|applicationContext\.xml|\/manager\/html|\/jmx-console\/|\.properties|\.class|phpinfo\.php|\/conn\.asp|\/conn\.php|\/conn\.jsp|config\.php" locahost -c


(19)第18檢測結果太多時,使用此精簡部分
grep -E -i "rootkit|3est|door|server|kim|phpspy|jspspy|command|shell|hack|f4ck|eval\(|system\(|\.jpg\.|AppScan|CustomCookie|netsparker|sqlmap|Havij|Pangolin|nessus|Openvas|whatWEB|w3af|DirBuster|WEBbench|OPTIONS|PUT|HEAD|DEBUG|system|action|login|manager|super|editor\/|(\w+)%27|(\w+)%20and%20(\S+)|(\w+)%20or%20(\S+)|(\d+)>(\d+)|(\d+)<(\d+)|(\S)waitfor(\W+)delay(\S)|(\S)having(\W)|(\S)sleep(\W)|(\w)--|(\S)select(\W)|(\S)insert(\S+)into(\W)|(\S)delete(\W)|(\S)update(\W)|(\S)create(\W)|(\S)drop(\W)|(\S)exists(\W)|(\S)backup(\W)|(\S)order(\S+)by(\W)|(\S)group(\S+)by(\W)|(\S)exec(\S)|(\S)truncate(\S)|(\S)declare(\S)|(\S)@@version(\S)|(\S)%3C(\S+)%3E|(\S)%3C(\S+)%2F%3E|(\S+)<(\S+)\/>|onerror|onmouse|expression|alert|document\.|prompt\(|\/etc\/passwd|\/%c0%ae%c0%ae|\/%2E%2E|boot\.ini|win\.ini|\.\.\/|access\.log|httpd\.conf|nginx\.conf|\/proc\/self\/environ|\/cmd\.asp|\/diy\.asp|\.asp;|\/(\w+)\.(\w+)\/(\w+)\.php|\.php\.|eval\(|%eval|\.jsp?action=|fsaction=|\/WEB-INF\/WEB\.xml|applicationContext\.xml|\/manager\/html|\/jmx-console\/|\.properties|\.class|phpinfo\.php|\/conn\.asp|\/conn\.php|\/conn\.jsp|config\.php" locahost -c

(20)檢視攻擊次數最多的IP
cat wapbank.log |awk '{print$3,$4,$7,$8}' | grep -E "\\s200\\s" | awk '{print $1}' | sort |uniq -c | sort -nr

(21)檢視攻擊型別排名
cat wapbank.log | grep -E "\\s200\\s" | awk '{print $2}' | sort |uniq -c | sort -nr >anttackType.txt

(22)檢視某IPURL排名
grep -E "106.38.128.101" access_log_edall.log |awk '{print$7}' | sort |uniq -c |sort -nr | more

(23)檢視某IPURL排序
grep -E "42.159.142.38" access_log_edall.log |sort | uniq -c | sort -nr |more

(24)檢視某IP返回200ok的資料包請求
grep -E "106.120.233.64" access_log_edall.log |sort | uniq -c | sort -nr |grep -E "\s200\s"

(25)檢視攻擊是否登陸後臺成功
grep -E "83.41.2.13" access_log_edall.log |sort | uniq -c | sort -nr |grep -E "\s200\s" |grep -E "manager|admin|login" -n |more