十一 kafka資料安全,以及Spark Kafka Streaming API
阿新 • • 發佈:2019-01-05
一基本網址 http://spark.apache.org/docs/1.6.2/api/java/index.html 在API中搜索org.apache.spark.streaming.kafka 二spark對接kafka流兩種方案 在org.apache.spark.streaming.kafka中有個KafkaUtils ,裡面有兩種方案 其中獲取kafka的流有兩種方案 2.1其中第一種 Modifier and type static <K,V,KD extends kafka.serializer.Decoder<K>,VD extends kafka.serializer.Decoder<V>,R> JavaInputDStream<R> Method and Description createDirectStream(JavaStreamingContext jssc, java.lang.Class<K> keyClass, java.lang.Class<V> valueClass, java.lang.Class<KD> keyDecoderClass, java.lang.Class<VD> valueDecoderClass, java.lang.Class<R> recordClass, java.util.Map<java.lang.String,java.lang.String> kafkaParams, java.util.Map<kafka.common.TopicAndPartition,java.lang.Long> fromOffsets, Function<kafka.message.MessageAndMetadata<K,V>,R> messageHandler) Create an input stream that directly pulls messages from Kafka Brokers without using any receiver. 2.2另外一種方案,方法簡單 static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> createStream(JavaStreamingContext jssc, java.lang.String zkQuorum, java.lang.String groupId, java.util.Map<java.lang.String,java.lang.Integer> topics) Create an input stream that pulls messages from Kafka Brokers. static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> createStream(JavaStreamingContext jssc, java.lang.String zkQuorum, java.lang.String groupId, java.util.Map<java.lang.String,java.lang.Integer> topics, StorageLevel storageLevel) Create an input stream that pulls messages from Kafka Brokers. 三網路文章 https://www.cnblogs.com/seaspring/p/5920414.html