Apache Eagle安裝配置指南
阿新 • • 發佈:2018-12-26
Apache Eagle開源沒多久,網上關於安裝它的文章幾乎沒有,在此我獻上自己的安裝步驟供大家參考。
官網安裝文件地址:http://goeagle.io/docs/deployment-in-production.html
官網的安裝大體步驟都沒問題,只是在配置的時候需要注意幾點。
1.環境依賴 HBase: 0.98 or later Storm: 0.9.3 or later Kafka: 0.8.x or later Zookeeper: 3.4.5 Java: 1.7.x Spark,用於使用者模型訓練 Logstash-1.5.5 2.Kafka配置 啟動kafka前需要先配置kafka中的zookeeper地址新建一個conf/Demo-hdfsAuditLog-application.conf# eagle configuration eagle{ # eagle web service configuration service{ # storage type: ["hbase","jdbc"] # default is "hbase" storage-type="hbase" # hbase configuration: hbase.zookeeper.quorum # default is "localhost" hbase-zookeeper-quorum="slave02.mesos,slave03.mesos,slave04.mesos" # hbase configuration: hbase.zookeeper.property.clientPort # default is 2181 hbase-zookeeper-property-clientPort=2181 # hbase configuration: zookeeper.znode.parent # default is "/hbase" zookeeper-znode-parent="/hbase" } }
注意配置這裡面的eagleService的host[email protected] conf]$ cat Demo-hdfsAuditLog-application.conf { "envContextConfig" : { "env" : "storm", "mode" : "cluster", "topologyName" : "Demo-hdfsAuditLog-topology", "stormConfigFile" : "security-auditlog-storm.yaml", "parallelismConfig" : { "kafkaMsgConsumer" : 1, "hdfsAuditLogAlertExecutor*" : 1 } }, "dataSourceConfig": { "topic" : "Demo_hdfs_audit_log", "zkConnection" : "slave02.mesos:2181,slave03.mesos:2181,slave04.mesos:2181", "zkConnectionTimeoutMS" : 15000, "fetchSize" : 1048586, "deserializerClass" : "eagle.security.auditlog.HdfsAuditLogKafkaDeserializer", "transactionZKServers" : "slave02.mesos,slave03.mesos,slave04.mesos", "transactionZKPort" : 2181, "transactionZKRoot" : "/consumers", "consumerGroupId" : "eagle.hdfsaudit.consumer", "transactionStateUpdateMS" : 2000 }, "alertExecutorConfigs" : { "hdfsAuditLogAlertExecutor" : { "parallelism" : 1, "partitioner" : "eagle.alert.policy.DefaultPolicyPartitioner", "needValidation" : "true" } }, "eagleProps" : { "site" : "Demo", "dataSource": "hdfsAuditLog", "dataJoinPollIntervalSec" : 30, "mailHost" : "tendcloud.com", "mailSmtpPort":"25", "mailDebug" : "true", "eagleService": { "host": "master.mesos", "port": 9099 "username": "admin", "password": "secret" } }, "dynamicConfigSource" : { "enabled" : true, "initDelayMillis" : 0, "delayMillis" : 30000 } }
4.啟動 建立hbase表 bin/eagle-service-init.sh 啟動eagle web service bin/eagle-service.sh start 引入metadata bin/eagle-topology-init.sh 5.登陸eagle web http://ip:9099/eagle-service 使用者名稱admin,密碼secret Storm中的topology
至於新增policy和site等過程請參見官網。 最後的web頁面如下: