用sbt構建Spark專案——WordCount
阿新 • • 發佈:2019-01-23
一、環境配置
1、sbt:http://www.scala-sbt.org/download.html 選擇windows的SBT-0.13.12.MSI即可,然後安裝
2、在系統環境中path後追加“sbt安裝目錄/bin”
3、用cmd進入本地命令窗,輸入“sbt”,等待jar包下載完成
4、進入C:\Users\***\.sbt\0.13\plugins,編輯plugins.sbt檔案,新增兩個外掛,程式碼如下
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0") addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
若無plugins資料夾和plugins.sbt,手動建立即可
二、用sbt建立eclipse專案
1、進入你的eclipse中的workspace資料夾,新建一個資料夾SbtWordCount,然後新建build.sbt檔案,輸入如下配置資訊
2、在命令列視窗中,進入到SbtWordCount目錄中,然後輸入“sbt eclipse”,等待出現Successfully created Eclipse ....就可以了name := "sbt-wordcount" version := "1.0" scalaVersion := "2.10.6" autoScalaLibrary := false EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.Resource EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.ManagedClasses libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.10" % "1.5.2" % "provided", "org.apache.spark" % "spark-mllib_2.10" % "1.5.2" % "provided", "org.apache.spark" % "spark-examples_2.10" % "1.1.1" % "provided" ) resolvers ++= Seq( // HTTPS is unavailable for Maven Central "Maven Repository" at "http://repo.maven.apache.org/maven2", "Apache Repository" at "https://repository.apache.org/content/repositories/releases", "JBoss Repository" at "https://repository.jboss.org/nexus/content/repositories/releases/", "MQTT Repository" at "https://repo.eclipse.org/content/repositories/paho-releases/", "Cloudera Repository" at "http://repository.cloudera.com/artifactory/cloudera-repos/", "le_bigdata_mining" at "http://10.150.144.28/nexus/content/repositories/releases/", Resolver.mavenLocal )
3、進入Scala IDE (eclipse)中,import該專案
4、若發現沒有src目錄,就手動建立。要想在windows下本地執行spark,還得需要winutils.exe,所以將其放在null/bin目錄中,可以參考上一篇文章windows中用scala-IDE開發spark—— WordCount