1. 程式人生 > >logstash+jdbc資料同步+head展示問題

logstash+jdbc資料同步+head展示問題

遇到問題

  • head UI只展示一條資料,索引過濾不了 - 原因: document_id => “%{user_id}” 中user_id 參考教程未改 - 解決:根據自己的主鍵改成物件的%{自己表的主鍵} - 後續:如果是多個表同步怎麼辦?
  • 同步時sql_last_value 一直是0,檢視logstash_jdbc_last_run也確實為0 - 原因: use_column_value=> false 是預設值,參照教程卻給設定成了true - 解決: 遮蔽use_column_value=> true
  • 示例同步檔案:
input{
         jdbc {
            # jdbc_driver_library => "mysql-connector-java-5.1.44-bin.jar"
    	# 驅動
             jdbc_driver_library => "/usr/local/soft/logstash-input-jdbc-master/lib/mysqljdbc-5.1.18.jar"
             jdbc_driver_class => "com.mysql.jdbc.Driver"
             jdbc_connection_string => "jdbc:mysql://172.16.0.178:3308/mtp_dev"
             jdbc_user => "mtp_dev"
             jdbc_password => "mtp_dev"
             jdbc_paging_enabled => "true"
             jdbc_page_size => "1000"
             jdbc_default_timezone =>"Asia/Shanghai"
             schedule => "* * * * *"
             statement => "SELECT * FROM user_sys_user where latest_time > :sql_last_value"
    	# record_last_run => true
            # use_column_value=> false
             tracking_column => "latest_time"
             last_run_metadata_path => "./logstash_jdbc_last_run"
             clean_run => "false"
           } 
    } 
    output{
          elasticsearch {
             hosts => "172.16.0.94:9200"
             user => "elastic"
             password => "changme"
             index => "sys_user"
             document_id => "%{user_id}"
          }
          stdout {
             codec => json_lines
         }
     }