windows下maven專案整合ELK演示
1、準備
1>jdk8及以上執行環境
java version "9.0.1"
2>下載Elasticsearch,Logstash,Kibana
https://www.elastic.co/downloads
2、安裝配置
1>elasticsearch-6.1.1\config\elasticsearch.yml增加下列配置：
http.cors.enabled: true
http.cors.allow-origin: "*"
備用dos命令：
netstat -ano|findstr "port"
檢視程序名： tasklist|findstr "PID"
2>測試：
windows+r輸入cmd 進入：你的安裝目錄\logstash-6.1.1\bin
你的安裝目錄\logstash-6.1.1\bin>logstash -f 'input { stdin { } } output { stdout {} }'
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 4
Logstash startup completed
echo hello world
{
"message" => "echo hello world\r",
"@version" => "1",
"@timestamp" => "2017-12-24T05:50:53.257Z",
"host" => "你的主機名"
}
3、使用
1>cmd執行elasticsearch-6.1.1：你的安裝目錄\elasticsearch-6.1.1\bin>elasticsearch.bat
2>在logstash的bin目錄新建logstash-test.conf
input {
tcp {
host => "127.0.0.1"
port => 8182
mode => "server"
ssl_enable => false
codec => json_lines
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "testIndex"
}
stdout { codec => rubydebug }
}
cmd執行logstash-6.1.1：你的安裝目錄\logstash-6.1.1\bin>logstash -f logstash-test.conf
這樣就正常
3>cmd執行kibana-6.1.1：你的安裝目錄\kibana-6.1.1-windows-x86_64\bin>kibana.bat
kibana 就是方便展示資料，不啟用也可以
4、maven專案
1>增加依賴包
<dependency>
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
<version>2.7.8</version>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.11</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>net.logstash.log4j</groupId>
<artifactId>jsonevent-layout</artifactId>
<version>1.7</version>
</dependency>
</dependencies>
2>日誌配置檔案增加
<!-- 傳送日誌到 logstash -->
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>127.0.0.1:8182</destination>
<!-- encoder is required -->
<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<root>
<level value="INFO" />
<appender-ref ref="LOGSTASH" />
</root>
3>類中使用 org.slf4j.LoggerFactory 列印日誌，如：
private static final Logger logger = LoggerFactory.getLogger(Test.class);
@Test
public void test2(){
logger.info("test3:","test3");
}
*********************************
啟動專案就會發送日誌到 127.0.0.1:8182 ，logstash一直監聽8182埠，有資料傳送來就會接收並插入elasticsearch（127.0.0.1:9200），生成logstash-test索引；就可以在kibana 中查看了。
啟動專案時，logstash中會顯示，這時就是在傳送日誌只是json解析異常
ES 的 DSL (domain specific language)一種JSON格式的查詢. 可以在cmd中：
curl "http://localhost:9200/logstash-test/_search?q=*&pretty"
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/query-dsl-query-string-query.html#query-string-syntax
5、遇到的問題及解決
1>版本最好保持一致；
2>這個不知道什麼原因，不影響；如有知道的可以告訴我
3.[2017-08-14T16:01:46,482][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"The setting `flush_size` in plugin `elasticsearch` is obsolete and is no longer available. This setting is no longer available as we now try to restrict bulk requests to sane sizes. See the 'Batch Sizes' section of the docs. If you think you still need to restrict payloads based on the number, not size, of events, please open a ticket. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}
解決：LogStash6及以上不支援：安裝nodeJS
npm install -g cnpm --registry=https://registry.npm.taobao.org
cnpm install --global gulp
npm install -g grunt-cli
npm install grunt --save-dev
npm install grunt-contrib-clean grunt-contrib-concat grunt-contrib-watch grunt-contrib-connect grunt-contrib-copy grunt-contrib-jasmine
4.ElasticSearch測試
資料 https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html
其他資料網上都有；
1>建立索引
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by"
:{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}
解決："{"""
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}curl: (7) Failed to connect to port 80: Connection refused
增加： -H "Content-Type:application/json;charset=UTF-8"
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [speaker]"}],"type":"mapper_parsing_exceptio
n","reason":"Failed to parse mapping [_default_]: No handler for type [string] declared on field [speaker]","caused_by":{"type":"mapper_parsing_exception","reas
on":"No handler for type [string] declared on field [speaker]"}},"status":400}
解決： packetbeat https://segmentfault.com/a/1190000008897731
String欄位被拆分成兩種新的資料型別: text用於全文搜尋的, 而keyword用於關鍵詞搜尋.
ES支援的型別有：String, date, long, double, boolean , ip, object, nested, geo_point, geo_shape, completion.
正確寫法：
curl -H "Content-Type:application/json;charset=UTF-8" -XPUT "http://localhost:9200/shakespeare/emp/1" -d "{"""mappings""" : {"""_default_""" : {"""properties""" : {"""speaker""" : {"""type""": """String""", """index""" : """not_analyzed""" },"""play_name"""
: {"""type""": """String""", """index""" : """not_analyzed""" },"""line_id""" : { """type""" : """integer""" },"""speech_number""" : { """type""" : """integer""" }}}}}"
或者
curl -H "Content-Type:application/json;charset=UTF-8" -XPUT "localhost:9200/mycompany/employee/1" -d "{\"first_name\":\"John\",\"last_name\":\"Smith\",\"age\":25,\"about\":\"Ilovetogorockclimbing\",\"interests\":[\"sports\",\"music\"]}" ;
2>匯入資料
curl -H "Content-Type:application/json;charset=UTF-8" -XPOST "http://localhost:9200/account/_bulk?pretty" --data-binary @accounts.json
curl -H "Content-Type:application/json;charset=UTF-8" -XPOST "http://localhost:9200/shakespeare/emp/_bulk?pretty" --data-binary @shakespeare_6.0.json
curl -H "Content-Type:application/json;charset=UTF-8" -XPOST "http://localhost:9200/_bulk?pretty" --data-binary @logs.jsonl
在Elasticsearch內部，對時間型別欄位，是統一採用 UTC 時間。在做查詢和顯示是需要轉換時間內容增加8個小時
檢視是否插入日誌： curl http://localhost:9200/shakespeare/_search?pretty
5、Elasticsearch-head
6以上安裝：https://www.cnblogs.com/xing901022/p/6030296.html