對RDD分區的理解
阿新 • • 發佈:2017-10-20
blog cat success ica sim contex mil ont eight
舉個例子:
val logFile = "file:///home/soyo/桌面/6.txt" val conf = new SparkConf().setAppName("Simple Application") val sc = new SparkContext(conf)
val RDD =sc.textFile(logFile)
RDD.saveAsTextFile("file:////home/soyo/桌面/667.txt")
val repartitionedRDD=context.repartition(8) //控制RDD分區的個數
repartitionedRDD.saveAsTextFile("file:////home/soyo/桌面/668.txt")
667.txt:
part-00000 _SUCCESS
668.txt:
part-00000 part-00001 part-00002 part-00003 part-00004 part-00005 part-00006 part-00007 _SUCCESS
如果設置.setMaster("local[2]")默認最少是2個分區,別的類同
對RDD分區的理解