sparksql語法,通過對映方式讀txt
阿新 • • 發佈:2019-01-06
Inferring the Schema Using Reflection(通過對映指定schema) --宣告一個類 scala> case class Person(name: String, age: Int) defined class Person --讀取txt檔案內容,然後對映到類的欄位,轉化成dataframe scala> val people = sc.textFile("hdfs://node1:8020/test/input/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt)).toDF() people: org.apache.spark.sql.DataFrame = [name: string, age: int] --變數註冊成表,表名為people scala> people.registerTempTable("people"); --執行sql scala> val teenagers=sqlContext.sql("select name,age from people where age>=13 and age<=19") teenagers: org.apache.spark.sql.DataFrame = [name: string, age: int] scala> teenagers.show +------+---+ | name|age| +------+---+ |Justin| 19| +------+---+ scala> teenagers.select("name").show +------+ | name| +------+ |Justin| +------+ scala> teenagers.count res19: Long = 1 --通過位置訪問 scala> teenagers.map(t => "Name: " + t(0)).collect().foreach(println) Name: Justin --通過欄位訪問 scala> teenagers.map(t => "Name: " + t.getAs[String]("name")).collect().foreach(println) Name: Justin --轉化成map對 scala> teenagers.map(_.getValuesMap[Any](List("name", "age"))).collect().foreach(println) Map(name -> Justin, age -> 19)