Spark元件之Spark Streaming學習6--如何呼叫Dstream裡面的getOrCompute方法?
阿新 • • 發佈:2019-01-27
1解釋
下圖中有getOrCompute在
在Dstream中有對getOrCompute的定義,但是是 private[streaming] 的,所以需要在streaming包下才能呼叫
:
private[streaming] final def getOrCompute(time: Time): Option[RDD[T]]
2.程式碼:
package org.apache.spark.streaming
import org.apache.spark.SparkConf
/**
* Created by xubo on 2016/5/22.
*/
object testDstream {
def main(args: Array[String]) {
val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
val ssc = new StreamingContext(conf, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
// words.getOrCompute(time) //time:Time
ssc.stop()
}
}
3.結果:
在streaming報下新建類,生成Dstream之後就可以呼叫 getOrCompute了