搭建ELK叢集 實時收集 jpress 專案日誌
阿新 • • 發佈：2020-08-18
ELK介紹
1.什麼是ELK
ELK是三個軟體
1.E：elasticsearch java程式 儲存，查詢日誌
2.L: logstash java程式 收集、過濾日誌
3.K: kibana java程式 提供web服務，將資料頁面化
4.F: filebeat go 收集、過濾日誌
2.ELK作用
1.收集： 收集所有伺服器的日誌
2.傳輸： 把日誌穩定的傳輸到ES或者其他地方
3.儲存： ES能有效快速的儲存日誌資料
4.分析： 通過web頁面分析資料
5.監控： 監控叢集架構
3.ELK優點
1.處理方式靈活：elasticsearch是實時全文索引，具有強大的搜尋功能 2.配置相對簡單：elasticsearch全部使用JSON 介面，logstash使用模組配置，kibana的配置檔案部分更簡單。 3.檢索效能高效：基於優秀的設計，雖然每次查詢都是實時，但是也可以達到百億級資料的查詢秒級響應。 4.叢集線性擴充套件：elasticsearch和logstash都可以靈活線性擴充套件 5.前端操作絢麗：kibana的前端設計比較絢麗，而且操作簡單
4.為什麼使用ELK
### ELK 叢集服務版本必須一致
#收集所有的日誌
web服務日誌
業務服務日誌
系統日誌
#統計、分析：
1.統計訪問量
2.統計訪問量前10的IP
3.站點訪問次數最多的URL
4.查詢一上午以上三個值
5.查詢一下午以上三個值
6.對比一下上下午使用者訪問量
7.對比這一週，每天使用者增長還是減少
環境準備
實現功能
1. tomcat部署jpress專案
2. nginx代理jpress
3. 實時獲取tomcat json格式日誌
4. 實時獲取nginx json格式日誌
5. 實現 redis 訊息佇列
| 主機名 | IP | 服務 | 環境要求 | 記憶體 |
|---|---|---|---|---|
| jojo01 | 10.0.0.20 | tomcat nginx logstash MySQL Redis | JDK | 4G |
| jojo02 | 10.0.0.21 | es kibana | JDK | 2G |
| jojo03 | 10.0.0.22 | es | JDK | 2G |
ES叢集部署
1.時間同步 yum install -y ntpdate ntpdate time1.aliyun.com 2.安裝Java環境 rz jdk-8u181-linux-x64.rpm rpm -ivh jdk-8u181-linux-x64.rpm 3.安裝ES rz elasticsearch-6.6.0.rpm #下載地址：https://www.elastic.co/downloads/elasticsearch rpm -ivh elasticsearch-6.6.0.rpm # 根據提示繼續操作 systemctl daemon-reload systemctl enable elasticsearch.service systemctl start elasticsearch.service 4.配置ES # 配置檔案 vim /etc/elasticsearch/elasticsearch.yml # jojo02 cluster.name: escluster node.name: es1 path.data: /service/es/data path.logs: /service/es/logs bootstrap.memory_lock: true bootstrap.system_call_filter: false http.port: 9200 transport.tcp.port: 9300 transport.tcp.compress: true network.host: 10.0.0.21,127.0.0.1 discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.unicast.hosts: [10.0.0.21","10.0.0.22"] # 叢集ip # jojo02 cluster.name: escluster node.name: es1 path.data: /service/es/data path.logs: /service/es/logs bootstrap.memory_lock: true bootstrap.system_call_filter: false http.port: 9200 transport.tcp.port: 9300 transport.tcp.compress: true network.host: 10.0.0.21,127.0.0.1 discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.unicast.hosts: [10.0.0.21","10.0.0.22"] # 叢集ip # 建立資料目錄 mkdir /service/es/{data,logs} -p chown -R elasticsearch.elasticsearch /service/es/ 5.配置啟動檔案中記憶體鎖 vim /usr/lib/systemd/system/elasticsearch.service [Service] LimitMEMLOCK=infinity 5.啟動ES systemctl daemon-reload systemctl start elasticsearch.service
Kibana 部署(jojo02)
1. 上傳程式碼包
rz kibana-6.6.0-x86_64.rpm
rpm -ivh kibana-6.6.0-x86_64.rpm
2. 配置kibana
vim /etc/kibana/kibana.yml
#程序的埠
server.port: 5601
#監聽地址
server.host: "10.0.0.21"
#指定ES的地址
elasticsearch.hosts: ["http://10.0.0.21:9200"]
#kibana也會建立索引
kibana.index: ".kibana"
3. 啟動kibana
systemctl start kibana.service
4. 訪問頁面
http://10.0.0.20:5601
Tomcat 部署(web01)
1. 時間同步
yum install -y ntpdate
ntpdate time1.aliyun.com
2. 安裝Java環境
rz jdk-8u181-linux-x64.rpm
rpm -ivh jdk-8u181-linux-x64.rpm
3.tomcat
yum install tomcat tomcat-webapps tomcat-admin-webapps -y
4.掛載wer包
# 下載地址
https://gitee.com/fuhai/jpress/blob/alpha/wars/jpress-web-newest.war
cd /usr/share/tomcat/webapps/ rz
4.啟動
systemctl start tomcat
# 修改tomcat 預設頁面
cp -rp /usr/share/tomcat/webapps/ /usr/share/tomcat/webapps-jpress
cd /usr/share/tomcat/webapps-jpress
rm -rf ROOT/*
mv jpress-v3.2.5/* ROOT/
# 修改預設地址及json格式
vim /usr/share/tomcat/conf/server.xml
<Service name="webapps-jpress">
<Engine name="webapps-jpress" resourceName="UserDatabase"/>
<Host name="localhost" appBase="webapps-jpress"
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat_access_json" suffix=".log"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
部署MySQL
1. 安裝依賴
yum install -y ncurses-devel libaio-devel gcc gcc-c++ glibc cmake autoconf openssl openssl-devel
2.上傳
rz
tar xf mysql-5.6.46-linux-glibc2.12-x86_64.tar.gz
3.建立目錄
mkdir /service
4.軟連結
mv mysql-5.6.46-linux-glibc2.12-x86_64 /service/
ln -s /service/mysql-5.6.46-linux-glibc2.12-x86_64 /service/mysql
5.建立使用者
useradd mysql -s /sbin/nologin -M
6.拷貝配置檔案和啟動指令碼
cd /service/mysql/support-files/
cp my-default.cnf /etc/my.cnf
cp: overwrite '/etc/my.cnf'? y
# 主庫配置檔案
vim /etc/my.cnf
[mysqld]
basedir = /service/mysql
datadir = /service/mysql/data
port=mysql
server_id=1
skip_name_resolve
log_err=/service/mysql/data/mysql.err
log_bin=/service/mysql/data/mysql-bin
cp mysql.server /etc/init.d/mysqld
7. 初始化
cd /service/mysql/scripts/
./mysql_install_db --user=mysql --basedir=/service/mysql --datadir=/service/mysql/data
8.system管理
vim /usr/lib/systemd/system/mysqld.service
[Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=https://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/service/mysql/bin/mysqld --defaults-file=/etc/my.cnf
LimitNOFILE = 5000
systemctl daemon-reload
systemctl start mysqld
9.新增環境變數
vim /etc/profile.d/mysql.sh
export PATH=/service/mysql/bin:$PATH
source /etc/profile
10.操作資料庫
mysql
create database php;
grant all on php.* to 'lyw' identified by '123';
Nginx 部署(web01)
1. 更換官方源
vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
2. 安裝nginx
yum install -y nginx
3.修改json格式日誌
vim /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
access_log /var/log/nginx/access.log json;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
4. 代理配置
vim /etc/nginx/conf.d/wp.conf
server {
listen 80;
server_name jpress.com;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
5. 啟動
systemctl start nginx
Logstash 部署(jojo01)
1. 時間同步
yum install -y ntpdate
ntpdate time1.aliyun.com
2. 安裝Java環境
rz jdk-8u181-linux-x64.rpm
rpm -ivh jdk-8u181-linux-x64.rpm
3. 安裝Logstash
rz logstash-6.6.0.rpm
rpm -ivh logstash-6.6.0.rpm
4. 授權
chown -R logstash.logstash /usr/share/logstash/
# nginx
/var/log/nginx/
access.log # json 格式
# tomcat
/usr/local/tomcat/logs/
tomcat_access # json 格式
vim /etc/logstash/conf.d/jpress_json.conf
input {
file {
type => "tomcat_access_json"
path => "/usr/share/tomcat/logs/tomcat_access_json*.log"
start_position => "beginning"
codec => "json"
}
}
input {
file {
type => "nginx_log_json"
path => "/var/log/nginx/access.log"
start_position => "beginning"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "%{type}_%{+YYYY-MM-dd}"
codec => "json"
}
}
# 啟動
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/jpress_json.conf &
實現 Reids 訊息佇列
yum install -y redis
vim /etc/redis.conf
bind 172.16.1.20 # redis 網路埠設定
systemctl start redis
vim /etc/logstash/conf.d/redis_json.conf
input {
file {
type => "nginx_log"
path => "/var/log/nginx/access.log"
start_position => "beginning"
codec => "json"
}
file {
type => "tomcat_log"
path => "/usr/share/tomcat/logs/tomcat_access_json.*.log"
start_position => "beginning"
codec => "json"
}
}
output {
if [type] == "nginx_log" {
redis {
host => "172.16.1.20"
port => "6379"
data_type => "list"
db => "0"
key => "nginx_log"
}
}
if [type] == "tomcat_log" {
redis {
host => "172.16.1.20"
port => "6379"
data_type => "list"
db => "1"
key => "tomcat_log"
}
}
}
# 啟動
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat_redis.conf &
# 檢視
172.16.1.20:6379> SELECT 1
OK
172.16.1.20:6379[1]> KEYS *
1) "tomcat_log"
172.16.1.20:6379[1]> KEYS *
1) "tomcat_log"
172.16.1.20:6379[1]> SELECT 0
OK
172.16.1.20:6379> KEYS *
1) "nginx_log"
實現 Reids 訊息佇列 到es叢集
vim /etc/logstash/conf.d/redis_to_es.conf
input {
redis {
host => "172.16.1.20"
port => "6379"
db => "0"
data_type => "list"
key => "nginx_log"
}
redis {
host => "172.16.1.20"
port => "6379"
db => "1"
data_type => "list"
key => "tomcat_log"
}
}
output {
if [type] == "nginx_log" {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "nginx_log_%{+YYYY-MM-dd}"
}
}
if [type] == "tomcat_log" {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "tomcat_log_%{+YYYY-MM-dd}"
}
}
}
# 啟動多例項
mkdir /data/logstash/redis_es
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_to_es.conf --path.data=/data/logstash/redis_es &
# 檢視 (瞬間被消費)
172.16.1.20:6379> KEYS *
(empty list or set)
172.16.1.20:6379>