docker-compose安裝kafka叢集及管理監控工具
編寫docker-compose檔案配置zk 和kafka叢集
#vim kafka.yml
version: '2' services: zoo1: image: zookeeper restart: always container_name: zoo1 ports: - "2181:2181" volumes: - "/data/zookeeper/zookeeper1/data:/data" - "/data/zookeeper/zookeeper1/datalog:/datalog" environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 zoo2: image: zookeeper restart: always container_name: zoo2 ports: - "2182:2181" volumes: - "/data/zookeeper/zookeeper2/data:/data" - "/data/zookeeper/zookeeper2/datalog:/datalog" environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 zoo3: image: zookeeper restart: always container_name: zoo3 ports: - "2183:2181" volumes: - "/data/zookeeper/zookeeper3/data:/data" - "/data/zookeeper/zookeeper3/datalog:/datalog" environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 broker1: image: wurstmeister/kafka restart: always container_name: broker1 ports: - "9091:9092" depends_on: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 1 KAFKA_ADVERTISED_HOST_NAME: broker1 KAFKA_ADVERTISED_PORT: 9092 KAFKA_HOST_NAME: broker1 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_LISTENERS: PLAINTEXT://broker1:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092 volumes: - /var/run/docker.sock:/var/run/docker.sock - "/data/kafka/kafka1/:/kafka" broker2: image: wurstmeister/kafka restart: always container_name: broker2 ports: - "9092:9092" depends_on: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 2 KAFKA_ADVERTISED_HOST_NAME: broker2 KAFKA_ADVERTISED_PORT: 9092 KAFKA_HOST_NAME: broker2 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_LISTENERS: PLAINTEXT://broker2:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092 volumes: - /var/run/docker.sock:/var/run/docker.sock - "/data/kafka/kafka2/:/kafka" broker3: image: wurstmeister/kafka restart: always container_name: broker3 ports: - "9093:9092" depends_on: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 3 KAFKA_ADVERTISED_HOST_NAME: broker3 KAFKA_ADVERTISED_PORT: 9092 KAFKA_HOST_NAME: broker3 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_LISTENERS: PLAINTEXT://broker3:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092 volumes: - /var/run/docker.sock:/var/run/docker.sock - "/data/kafka/kafka3/:/kafka" |
執行compose檔案
| #COMPOSE_PROJECT_NAME=kafkatest docker-compose -f kafka.yml up -d |
檢視docker啟動程序
| [[email protected] ~]# docker ps |
zk 和kafka已啟動
kafka-manager管理工具安裝
#vim kafka_manager.yml |
version: '3.1' services: kafka-manager: container_name: kafka-manager image: sheepkiller/kafka-manager restart: always ports: - 9000:9000 environment: KM_VERSION: 1.3.3.18 ZK_HOSTS: zoo1:2181,zoo2:2182,zoo3:2183 |
啟動docker容器
| docker-compsoe -f kafka-manager.yml up -d |
docker啟動後，登陸到docker容器裡
| #vim conf/application.conf |
kafka-manager.zkhosts="zoo1:2181,zoo2:2182,zoo3:2183" #kafka-manager.zkhosts=${?ZK_HOSTS} |
將配置檔案注掉一條，然後將zk的地址寫入，不然介面是出不來的
以上是出現如下錯誤的解決方法
Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [5000 ms]
kafka-manager不支援硬碟掛載，我也被坑了好久，
kafkaoffsetmonitor監控工具安裝配置
#vim kafkaoffsetmonitor.yml
vim kafkaoffsetmonitor.yml version: '2' services: kafka-offset-monitor: image: 564239555/kafkaoffsetmonitor volumes: - /data/kafkaoffsetmonitor/conf:/kafkaoffsetmonitor ports: - "8089:8089" environment: ZK_HOSTS: zoo1:2181,zoo2:2182,zoo3:2183 KAFKA_BROKERS: 172.16.70.51:9091,172.16.70.51:9092,172.16.70.51:90093 REFRESH_SECENDS: 10 RETAIN_DAYS: 2 |
啟動映象
| docker-compose -f kafkaoffsetmonitor.yml up -d |
kafkaoffsetmonitor需要一個檔案來啟動，KafkaOffsetMonitor-assembly-0.2.1.jar，我已將些檔案放到映象中
或者自行下載這個軟體，將KafkaOffsetMonitor-assembly-0.2.1.jar 拷貝到docker 容器裡，
docker cp /data/kafkaoffsetmonitor/conf/KafkaOffsetMonitor-assembly-0.2.1.jar a45b7987f267:/kafkaoffsetmonitor/ |
寫個啟動指令碼
#vim /kafkaoffsetmonitor/start.sh
#!/bin/bash nohup java -cp KafkaOffsetMonitor-assembly-0.2.1.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk zoo1:2181,zoo2:2182,zoo3:2183 --port 8089 --refresh 10.seconds --retain 2.days & 如果啟動報錯，將--zk 改成宿主機IP |
給執行許可權然後啟動指令碼
#chmod +x /kafkaoffsetmonitor/start.sh #sh /kafkaoffsetmonitor/start.sh |
看到8089的埠啟動，就可以通過瀏覽器http:IP:8089去訪問
網上很多映象沒有這個檔案，啟動就報錯
或者直接下載我的映象檔案
https://hub.docker.com/r/564239555/kafkaoffsetmonitor/
| docker pull 564239555/kafkaoffsetmonitor |
docker啟動後，登陸到docker容器裡，執行指令碼，看到8089的埠啟動就可以通過瀏覽器去訪問了http://宿主機IP:8089
| #docker exec -it $dockerCONTAINER ID bash |
修改docker啟動所需要的指令碼檔案
#cd /kafkaoffsetmonitor/conf/ 需要將zk更改成你自已叢集的IP 地址 #./kafkaoffsetmonitor.sh #ss -tunl |