在Linux環境下部署confluent platform
1. confluent platform的簡單介紹:
confluent platform是一個可靠、高效能的流平臺,提供組織和管理來自不同資料來源資料的功能。(這個說法太正式,反正我是簡單粗暴地把它當成kafka和一些擴充套件kafka功能的一套元件~)
2. confluent platform的部署流程:
廢話不多說,直接開搞~
(1) 安裝JDK(建議安裝1.8的)
這個很簡單,就跳過吧,不會的請自行度娘~
(2) 配置主機名和ip的對映(zookeeper的主機之間是使用主機名相互通訊的)
vim /etc/hosts
192.168.20.240 host240
192.168.20.239 host239
192.168.20.238 host238
(ps: zookeeper至少需要三臺機子,然後改hosts需要root許可權~)
(3) 安裝zookeeper
雖然confluent platform自帶一個,但是還是推薦自己裝一個更高版本的~
a. 下載
b. 解壓後進入目錄
tar -zxf zookeeper-3.4.11.tar.gz
cd zookeeper-3.4.11
c. 新增配置檔案
複製一份conf/zoo_sample.cfg,命名為zoo.cfg
cp conf/zoo_sample.cfg conf/zoo.cfg
我的配置(修改的會有中文說明)如下:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
#zookeeper資料目錄
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
#根據機子數目做相應的配置,server.x中的x是下面myid需要的
server.240=host240:2888:3888
server.239=host239:2888:3888
server.238=host238:2888:3888
(ps:這個配置每臺機子都一樣~)
d. 新增myid檔案
在上一步配置的zookeeper資料目錄下新建檔案myid,加入上一步對應配置的x(比如host240對應配置的是240)
cat /data/zookeeper/myid
240
e. 啟動zookeeper
分別在三臺主機上啟動
bin/zkServer.sh start
(4) confluent platform安裝
a. 下載
(ps: 選擇下載開源版本,郵件可以亂填,只要能通過郵件的正則校驗即可,不用登陸郵件確認~)
b. 解壓後進入目錄
tar -zxf confluent-oss-4.0.0-2.11.tar.gz
cd confluent-oss-4.0.0
c. 修改配置檔案
至少需要配置etc/kafka目錄下的server.properties,我的配置(修改的會有中文說明)如下:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
# kafka broker id,必須唯一
broker.id=240
# Switch to enable topic deletion or not, default value is false
# 這個預設是false,不過建議開啟方便刪除~
delete.topic.enable=true
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
# kafka 監聽埠,預設是9092
listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma seperated list of directories under which to store log files
# kafka 日誌儲存路徑
log.dirs=/data/gameserver/bigdata/confluent
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
# 預設分割槽數(建立topic不指定分割槽數目時)
num.partitions=32
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
# 日誌儲存時長
log.retention.hours=48
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
# zookeeper地址(沒有/confluent的話會儲存在zookeeper的根目錄~)
zookeeper.connect=host240:2181,host239:2181,host238:2181/confluent
# Timeout in ms for connecting to zookeeper
# zookeeper超時時間
zookeeper.connection.timeout.ms=6000
##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics kakfa cluster.
#metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
#confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
#confluent.metrics.reporter.topic.replicas=1
##################### Confluent Proactive Support ######################
# If set to true, and confluent-support-metrics package is installed
# then the feature to collect and report support metrics
# ("Metrics") is enabled. If set to false, the feature is disabled.
confluent.support.metrics.enable=false
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
# The customer ID under which support metrics will be collected and
# reported.
#
# When the customer ID is set to "anonymous" (the default), then only a
# reduced set of metrics is being collected and reported.
#
# Confluent customers
# -------------------
# If you are a Confluent customer, then you should replace the default
# value with your actual Confluent customer ID. Doing so will ensure
# that additional support metrics will be collected and reported.
#
confluent.support.customer.id=anonymous
這裡只展示kafka的配置,其他元件的配置也都在etc目錄下,根據自己的需要選擇相應的配置即可~
(ps:別忘了host239和host238兩臺機子也需要做相同的配置,除了broker.id之外~)
d. 啟動
啟動指令碼都在bin目錄下面,分別在三臺機子上啟動命令
nohup ./bin/kafka-server-start ./etc/kafka/server.properties &
先通過Java的jps命令檢視是否存在SupportedKafka程序,再檢視logs/server.log是否有報錯資訊。
(ps:其他的元件基本上都是修改配置,然後用上面類似的啟動命令啟動,就不一一展示了~)
e. 測試
如果啟動沒問題了,可以試著在一臺機子新建topic
bin/kafka-topics --create --zookeeper host240:2181,host239:2181,host238:2181/confluent --replication-factor 1 --partitions 1 --topic test
然後在其他機子檢視是否能看到新建的topic
bin/kafka-topics --list --zookeeper host240:2181,host239:2181,host238:2181/confluent
如果能看到的話基本就算裝好了,可以試著往裡面寫點資料
bin/kafka-console-producer --broker-list host240:9092,host239:9092,host238:9092 --topic test
This is a message
然後讀取看看
bin/kafka-console-consumer --bootstrap-server host240:9092,host239:9092,host238:9092 --topic test --from-beginning
如果能讀取到寫入的資料,kafka就算啟動成功了。
相關推薦
在Linux環境下部署confluent platform
1. confluent platform的簡單介紹: confluent platform是一個可靠、高效能的流平臺,提供組織和管理來自不同資料來源資料的功能。(這個說法太正式,反正我是簡單粗暴地把它當成kafka和一些擴充套件kafka功能的一套元件~)
linux環境下部署zabbix3.2、模板、郵件告警詳細過程
-1 ice erer without zlib zip ever native item 服務端部署: 系統環境及軟件版本: Linux:release 6.3 zabbix:zabbix-3.2.5.tar.gz nginx:nginx-1.12.0.tar.gz ph
JAVA(JDK)在linux環境下部署及配置環境變數
1.首先在網上下載一個jdk的壓縮檔案,以gz為結尾的 2.開啟機器。cd /usr/local 到這個目錄下,rz上傳檔案 3.等待檔案上傳好後,ll檢視一下,然後建立一個資料夾mkdir -p /usr/java。之後把jdk檔案移動到java目錄下。mv
Linux環境下部署B/S應用測試環境
1.替換war包: 登入Fillzilla(FTP工具),輸入主機、使用者名稱、密碼、埠建立連線,進入tomcat-webapps下,刪除linux環境下的測試包,或是直接使用本地的測試包覆蓋原來的測試包(備份原來的測試war包,使用命令 mv file fil
.NetCore~Linux環境下部署
回到目錄 NetCore正式版已經出現有段時候了,Windows下使用vs2015開發.netCore應用程式,然後通過dotnet程式開啟WEB服務,用著很像node.js,當然我們不會於只侷限於windows,.netCore自出生以來,就是為了跨平臺而引入了,Linux是一個更穩定和通用的生產環境,下
Linux環境下部署TestLink指導手冊--3
安裝TestLink 下載 安裝 首先修改/opt/lampp/htdocs/testlink/config.inc.php檔案中以下兩個目錄地址: $tlCfg->log_path = '/opt/lampp/htdocs/testlin
Linux環境下部署TestLink指導手冊--2
安裝Maintis 下載 安裝 在本地電腦登陸安裝頁面:http://192.168.10.152:8080/mantis/ 按照介面提示一步步安裝即可,資料庫名稱設定為mantis,使用者名稱密碼一律root,123456。 報錯config_i
Linux環境下部署XX-Net
所需工具:XX-Net,Chrome瀏覽器(1)下載XX-Net:下載連結(2)開啟IPV-61.安裝虛擬網絡卡sudo apt-get install miredo2.檢視是否安裝成功ifconfig(存在一個叫teredo)的網絡卡3.啟動Miredosudo mired
Linux環境下部署Dubbo 監控中心
1.安裝JDK 2.安裝Tomcat 3.部署監控中心:[[email protected] ~]# cp dubbo-admin-2.5.4.war apache-tomcat-7
在linux環境下部署tomcat應用
配置linux下的tomcat一、jdk的安裝1.以root身份等陸 2.開啟一個終端 3.輸入命令ls -l檢視安裝的檔案的許可權 #cd /home #cp jdk1.4.2.rpm.bin /usr/local #cd /usr/local 給所有的使用者新增可執行的許
linux環境下部署tomcat工程,日誌不走log4j配置而列印到catalina.out可能的原因
平時不積累,出現在問題時就麻爪orz 問題描述:java web應用部署後,出現日誌不走配置的log4j日誌列印目錄,而是走tomcat的catalina.out,web應用配置沒問題。 錯誤思路:部署後配置未生效,重新部署後還是存在問題,本地環境卻沒問題 正確思路:lin
kettle-linux環境下部署kettle,執行kitchen.sh檔案報錯後安裝 libwebkitgtk,提示沒有可用軟體包libwebkitgtk。
執行./kitchen.sh 報錯資訊: [[email protected] data-integration]# ./kitchen.sh ##########################################################
linux環境下jdk部署配置
etc $path 是否 ssp java jdk1 配置系統 exp 執行文件 1、java官網下載相關的jdk包 2、配置系統環境變量,編輯/etc/profile文件,在文件的末尾添加一下信息: export JAVA_HOME=/usr/jdk1.8.0_101ex
在Linux環境下使用Apache部署ASP.NET Core
tgt tab dot sce aaa target 部署 AI startup 在前幾篇文章中我們一起探討了如何在Linux環境中安裝ASP.NET Core運行時環境及將ASP.NET Core項目部署在Jexus中,這篇文章中我們將探討如何將ASP.NET Core部
3. linux centos6.8環境下部署zookeeper集群
path port etc amp 配置ip地址 指定 leader 路徑 ftw 安裝3臺機器來作為zookeeper集群,3臺機器的ip分別是192.168.174.7,192.168.174.8,192.168.174.9。zookeeper由java編寫,所以在安裝
測試環境搭建和部署(在Linux環境下搭建jdk+Tomcat+mysql環境和項目包的部署)
pri 環境 安裝包 mysq 自己 進行 配置jdk linu 連接數 1.Linux環境(我搭建的是64位centos版本的linux系統) 1.下載並安裝一個VMware Workstation虛擬機,是搭建Linux系統的平臺。 2.下載一個
部署openfire到linux環境下
1.java環境部署:具體參考 http://blog.csdn.net/gufachongyang02/article/details/453373792.ant環境部署: 具體參考 http://blog.csdn.net/gufachongyang02/article/details/453373793
[Linux] CentOS 下部署 Java Web 開發環境
安裝 JDK 環境 見另一篇部落格《CentOS 7 配置 Oracle JDK》。 安裝 Eclipse 使用 Eclipse 作為 Java Web 專案的開發 IDE。 從 Eclipse 官網下載 Eclipse 的 Linux 版本(例如 eclipse-inst
Redis在linux環境下的安裝和部署
官網:http://redis.io windows版本下載地址https://github.com/MicrosoftArchive/redis/releases 1Redis建議安
記錄mosquitto在Linux環境下的部署/安裝/使用/測試過程
Mosquitto mosquitto是一款實現了 MQTT v3.1 協議的開源的訊息代理服務軟體. 其提供了非常輕量級的訊息資料傳輸協議,採用釋出/訂閱模式進行工作,可用於物聯裝置、中介軟體、APP客戶端之間的訊息通訊。 mosquitto官網 關於mqtt協