1. 程式人生 > >十一 kafka資料安全,以及Spark Kafka Streaming API

十一 kafka資料安全,以及Spark Kafka Streaming API

一基本網址
http://spark.apache.org/docs/1.6.2/api/java/index.html
在API中搜索org.apache.spark.streaming.kafka

二spark對接kafka流兩種方案
在org.apache.spark.streaming.kafka中有個KafkaUtils ,裡面有兩種方案
其中獲取kafka的流有兩種方案
2.1其中第一種
Modifier and type
static <K,V,KD extends kafka.serializer.Decoder<K>,VD extends kafka.serializer.Decoder<V>,R>
JavaInputDStream<R>

Method and Description
createDirectStream(JavaStreamingContext jssc, java.lang.Class<K> keyClass, java.lang.Class<V> valueClass, java.lang.Class<KD> keyDecoderClass, java.lang.Class<VD> valueDecoderClass, java.lang.Class<R> recordClass, java.util.Map<java.lang.String,java.lang.String> kafkaParams, java.util.Map<kafka.common.TopicAndPartition,java.lang.Long> fromOffsets, Function<kafka.message.MessageAndMetadata<K,V>,R> messageHandler)

Create an input stream that directly pulls messages from Kafka Brokers without using any receiver.
2.2另外一種方案,方法簡單
static JavaPairReceiverInputDStream<java.lang.String,java.lang.String>
static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> createStream(JavaStreamingContext jssc, java.lang.String zkQuorum, java.lang.String groupId, java.util.Map<java.lang.String,java.lang.Integer> topics)
Create an input stream that pulls messages from Kafka Brokers.

static JavaPairReceiverInputDStream<java.lang.String,java.lang.String>
static JavaPairReceiverInputDStream<java.lang.String,java.lang.String> createStream(JavaStreamingContext jssc, java.lang.String zkQuorum, java.lang.String groupId, java.util.Map<java.lang.String,java.lang.Integer> topics, StorageLevel storageLevel)
Create an input stream that pulls messages from Kafka Brokers.

三網路文章
https://www.cnblogs.com/seaspring/p/5920414.html