flume的kafka到hdfs的配置
阿新 • • 發佈:2019-01-25
1.htrace-core-3.0.jar
2.commons-configuration-1.6.jar
3.hadoop-auth-2.6.0.jar
4.hadoop-common-2.6.0.jar
5.hadoop-hdfs-2.6.0.jar
6.hadoop-mapreduce-client-core-2.6.0.jar
遇到這些錯誤時,新增以上包就可以解決:
java.lang.NoClassDefFoundError:org/apache/commons/configuration/Configuration
java.lang.NoClassDefFoundError:org/apache/hadoop/util /PlatformName
java.io.IOException:No FileSystem forscheme:hdfs
java.io.IOException:Incomplete HDFS URI,no host:hdfs://
flume的conf配置:
agent.sources=r1 agent.sinks=k1 agent.channels=c1 agent.sources.r1.type= org.apache.flume.source.kafka.KafkaSource agent.sources.r1.kafka.bootstrap.servers=192.168.0.1:9092 agent.sources.r1.kafka.topics=TEST-TOPIC #agent.sources.r1.serializer.class=kafka.serializer.StringEncoder agent.sources.r1.kafka.consumer.group.id = flumetest auto.commit.enable = true agent.sources.r1.flumeBatchSize=1000 agent.sources.r1.useFlumeEventFormat=false agent.sources.r1.restart=true agent.sources.r1.batchSize=1000 agent.sources.r1.batchTimeout=3000 agent.sources.r1.channels=c1 agent.channels.c1.type=memory agent.channels.c1.capacity=102400 agent.channels.c1.transactionCapacity=1000 agent.channels.c1.byteCapacity=134217728 agent.channels.c1.byteCapacityBufferPercentage=80 agent.sinks.k1.channel=c1 agent.sinks.k1.type=hdfs agent.sinks.k1.hdfs.path=hdfs://master02:8020/tmp/flume/ds=%Y%m%d agent.sinks.k1.hdfs.rollSize=0 agent.sinks.k1.hdfs.rollCount=0 agent.sinks.k1.hdfs.rollInterval=3600 agent.sinks.k1.hdfs.threadsPoolSize=30 agent.sinks.k1.hdfs.fileType=DataStream agent.sinks.k1.hdfs.writeFormat=Text ## source 攔截器 agent.sources.r1.interceptors = i1 agent.sources.r1.interceptors.i1.type = org.com.flume.MyInterceptor$Builder