Logstash 安裝部署注意事項：

• Logstash基本概念:

• logstash收集日誌基本流程: input-->codec-->filter-->codec-->output

• input:從哪裡收集日誌。

• filter:發出去前進行過濾

• output:輸出至Elasticsearch或Redis訊息佇列

• codec:輸出至前臺，方便邊實踐邊測試

• 資料量不大日誌按照月來進行收集

• 如果通過logstash來採集日誌，那麼每個客戶端都需要安裝logstash

• 安裝需要前置系統環境

• 安裝java

• 下載並安裝GPG key [[email protected]_agent yum.repos.d]# rpm --import

  https://packages.elastic.co/GPG-KEY-elasticsearch

• 配置yum倉庫

[[email protected]_agentyum.repos.d]#echo'[logstash-2.3]>name=Logstashrepositoryfor2.3.xpackages
>baseurl=https://packages.elastic.co/logstash/2.3/centos>gpgcheck=1>gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
>enabled=1'>>/etc/yum.repos.d/logstash.repo

• 安裝logstash

[[email protected]_agentyum.repos.d]#yuminstall-ylogstash

• 檢視多種輸出輸出方式

例項1

[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'

• 輸出內容：

[[email protected]_agentyum.repos.d]#/opt/logstash/bin/logstash-e'input{stdin{}}output{stdout{}}'OpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone,thenyoushouldconfigurethenumberofparallelGCthreadsappropriatelyusing-XX:ParallelGCThreads=N
Settings:Defaultpipelineworkers:1Pipelinemainstarted
hello2017-06-27T04:10:21.010Zmaster_agenthello
helloword2017-06-27T04:10:48.513Zmaster_agenthellowordquit2017-06-27T04:11:12.959Zmaster_agentquitexit2017-06-27T04:11:19.064Zmaster_agentexit

例項2

[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'

[[email protected]_agentyum.repos.d]#/opt/logstash/bin/logstash-e'input{stdin{}}output{stdout{codec=>rubydebug}}'OpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone,thenyoushouldconfigurethenumberofparallelGCthreadsappropriatelyusing-XX:ParallelGCThreads=N
Settings:Defaultpipelineworkers:1Pipelinemainstarted
helloword
{"message"=>"helloword","@version"=>"1","@timestamp"=>"2017-06-27T04:13:14.560Z","host"=>"master_agent"}

例項3 不輸出在螢幕上，直接記錄在es裡

[[email protected]_agentyum.repos.d]#/opt/logstash/bin/logstash-e'input{stdin{}}output{elasticsearch{hosts=>["172.16.1.201:9200"]}}'OpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone,thenyoushouldconfigurethenumberofparallelGCthreadsappropriatelyusing-XX:ParallelGCThreads=N
Settings:Defaultpipelineworkers:1Pipelinemainstarted
helloword!!
Pipelinemainhasbeenshutdown

例項5

• 記錄兩份，螢幕輸出一份、es記錄一份 [[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } stdout{ codec => rubydebug } }'

[[email protected]_agentyum.repos.d]#/opt/logstash/bin/logstash-e'input{stdin{}}output{elasticsearch{hosts=>["172.16.1.201:9200"]}stdout{codec=>rubydebug}}'OpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone,thenyoushouldconfigurethenumberofparallelGCthreadsappropriatelyusing-XX:ParallelGCThreads=N
Settings:Defaultpipelineworkers:1Pipelinemainstarted
helloflysalt
{"message"=>"helloflysalt","@version"=>"1","@timestamp"=>"2017-06-27T04:32:50.297Z","host"=>"master_agent"}

檢視web頁面記錄的內容

Logstash配置檔案

• 配置Logstash,將建立一個配置檔案,指定您想要使用的外掛和設定為每個外掛。 你可以參考事件欄位在配置和使用條件符合一定時處理事件標準；當您執行logstash,你使用-f指定您的配置檔案。參考連結

• 例項如下：

• [[email protected]_agent yum.repos.d]# vim /etc/logstash/conf.d/logstash.conf ##新建一個配置語法檔案，讓輸入輸出標準化

input{stdin{}}output{
elasticsearch{hosts=>["localhost:9200"]}stdout{codec=>rubydebug}
}

• [[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf ### 執行命令配置語法檔案手動輸入hao123.com，檢視結果

[[email protected]_agentyum.repos.d]#/opt/logstash/bin/logstash-f/etc/logstash/conf.d/logstash.confOpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone,thenyoushouldconfigurethenumberofparallelGCthreadsappropriatelyusing-XX:ParallelGCThreads=N
Settings:Defaultpipelineworkers:1Pipelinemainstarted
hao123.com
{"message"=>"hao123.com","@version"=>"1","@timestamp"=>"2017-06-27T05:06:02.695Z","host"=>"master_agent"}

input外掛

官方連結

• 建立一個資料夾用於專門存放logfile配置檔案 存放的路徑

• [[email protected]_agent yum.repos.d]# mkdir -p /home/elk/logfile/

• Input plugins file 收集例項：

• 編輯一個新的配置檔案

• 
  

[[email protected]_agentyum.repos.d]#mkdir-p/home/elk/logfile/[[email protected]_agentyum.repos.d]#cd/home/elk/logfile/[[email protected]_agentlogfile]#ls[[email protected]_agentlogfile]#vimfile.conf

• 編寫配置檔案內容

input{
file{
path=>"/var/log/messages"###讀取log日誌檔案路徑
type=>"system"###設定個型別
start_position=>"beginning"###從日誌最開始地方讀取
}
}output{
elasticsearch{
hosts=>["172.16.1.201:9200"]
index=>"system-%{+YYYY.MM.dd}"##設定個索引每天建立
}

}

注意：如果採集的日誌檔案日誌量不是很大的情況下，採用每月建一從索引，日誌量很大的情況下可以按照每天來建立索引

轉載於:https://blog.51cto.com/yibeishui/1953414