Spring boot with ELK(Elasticsearch + Logstash + Kibana)
阿新 • • 發佈:2022-05-02
本文節選自《Netkiller Java 手札》
5.31. Spring boot with ELK(Elasticsearch + Logstash + Kibana)
將 Spring boot 日誌寫入 ELK 有多種實現方式,這裡僅提供三種方案:
- Spring boot -> logback -> Tcp/IP -> logstash -> elasticsearch 這種方式實現非常方便不需要而外包或者軟體
- Spring boot -> logback -> Redis -> logstash -> elasticsearch 利用 Redis 提供的釋出訂閱功能將日誌投遞到 elasticsearch
- Spring boot -> logback -> Kafka -> logstash -> elasticsearch Kafka 方法適合大資料的情況。
5.31.1. TCP 方案
logstash 配置
input { tcp { host => "172.16.1.16" port => 9250 mode => "server" tags => ["tags"] codec => json_lines //可能需要更新logstash外掛 } } output { stdout{codec =>rubydebug} elasticsearch { hosts => ["localhost:9200"] //這塊配置需要帶埠號 flush_size => 1000 } }
Spring boot logback.xml 配置
<?xml version="1.0" encoding="UTF-8"?> <configuration> <property resource="properties/logback-variables.properties" /> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder charset="UTF-8"> <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n </pattern> </encoder> </appender> <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>172.16.1.16:9250</destination> <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" /> </appender> <!--<appender name="async" class="ch.qos.logback.classic.AsyncAppender">--> <!--<appender-ref ref="stash" />--> <!--</appender>--> <root level="info"> <!-- 設定日誌級別 --> <appender-ref ref="STDOUT" /> <appender-ref ref="LOGSTASH" /> </root> </configuration>
5.31.2. Redis 方案
Maven pom.xml 增加 Logback Redis 依賴
<!-- https://mvnrepository.com/artifact/com.cwbase/logback-redis-appender -->
<dependency>
<groupId>com.cwbase</groupId>
<artifactId>logback-redis-appender</artifactId>
<version>1.1.5</version>
</dependency>
Spring boot logback.xml 配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<property name="type.name" value="test" />
<appender name="LOGSTASH" class="com.cwbase.logback.RedisAppender">
<source>spring-application</source>
<type>${type.name}</type>
<host>localhost</host>
<key>logstash:redis</key>
<tags>test-2</tags>
<mdc>true</mdc>
<location>true</location>
<callerStackIndex>0</callerStackIndex>
<!--additionalField新增附加欄位 用於head外掛顯示 -->
<additionalField>
<key>MyKey</key>
<value>MyValue</value>
</additionalField>
<additionalField>
<key>MySecondKey</key>
<value>MyOtherValue</value>
</additionalField>
</appender>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="LOGSTASH" />
</root>
</configuration>
logstash 配置
input {
redis {
host => 'localhost'
data_type => 'list'
port => "6379"
key => 'logstash:redis' #自定義
type => 'redis-input' #自定義
}
}
output {
elasticsearch {
host => "localhost"
codec => "json"
protocol => "http"
}
}
5.31.3. Kafka 方案
暫時沒有機器資源,本章節待續