Spark API 詳解/大白話解釋 之 groupBy、groupByKey
阿新 • • 發佈:2018-12-25
groupBy(function)
function返回key,傳入的RDD的各個元素根據這個key進行分組
val a = sc.parallelize(1 to 9, 3)
a.groupBy(x => { if (x % 2 == 0) "even" else "odd" }).collect//分成兩組
/*結果
Array(
(even,ArrayBuffer(2, 4, 6, 8)),
(odd,ArrayBuffer(1, 3, 5, 7, 9))
)
*/
val a = sc.parallelize(1 to 9, 3)
def myfunc(a: Int) : Int =
{
a % 2 //分成兩組
}
a.groupBy(myfunc).collect
/*
結果
Array(
(0,ArrayBuffer(2, 4, 6, 8)),
(1,ArrayBuffer(1, 3, 5, 7, 9))
)
*/
groupByKey( )
val a = sc.parallelize(List("dog", "tiger", "lion", "cat", "spider", "eagle"), 2)
val b = a.keyBy(_.length)//給value加上key,key為對應string的長度
b.groupByKey.collect
//結果 Array((4,ArrayBuffer(lion)), (6,ArrayBuffer(spider)), (3,ArrayBuffer(dog, cat)), (5,ArrayBuffer(tiger, eagle)))