Spark運算元:Action之first、count、reduce、collect
阿新 • • 發佈:2018-12-11
1、first:def first(): T
該函式返回RDD的第一個元素,不排序。
scala> var rdd1 = sc.makeRDD(Array(("A","1"),("B","2"),("C","3")),2) rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[33] at makeRDD at :21 scala> rdd1.first res14: (String, String) = (A,1) scala> var rdd1 = sc.makeRDD(Seq(10, 4, 2, 12, 3)) rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at makeRDD at :21 scala> rdd1.first res8: Int = 10
2、count:def count(): Long
該函式返回RDD中的元素總數量。
scala> var rdd1 = sc.makeRDD(Array(("A","1"),("B","2"),("C","3")),2)
rdd1: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[34] at makeRDD at :21
scala> rdd1.count
res15: Long = 3
3、reduce:def reduce(f: (T, T) ⇒ T): T
該函式根據對映函式f對RDD中的元素進行二元計算,返回計算結果。
scala> var rdd1 = sc.makeRDD(1 to 10,2) rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[36] at makeRDD at :21 scala> rdd1.reduce(_ + _) res18: Int = 55 scala> var rdd2 = sc.makeRDD(Array(("A",0),("A",2),("B",1),("B",2),("C",1))) rdd2: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[38] at makeRDD at :21 //字元相串,數字相加 scala> rdd2.reduce((x,y) => { | (x._1 + y._1,x._2 + y._2) | }) res21: (String, Int) = (CBBAA,6)