1. 程式人生 > >用sbt構建Spark專案——WordCount

用sbt構建Spark專案——WordCount

一、環境配置

1、sbt:http://www.scala-sbt.org/download.html    選擇windows的SBT-0.13.12.MSI即可,然後安裝

2、在系統環境中path後追加“sbt安裝目錄/bin”

3、用cmd進入本地命令窗,輸入“sbt”,等待jar包下載完成

4、進入C:\Users\***\.sbt\0.13\plugins,編輯plugins.sbt檔案,新增兩個外掛,程式碼如下

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

若無plugins資料夾和plugins.sbt,手動建立即可

二、用sbt建立eclipse專案

1、進入你的eclipse中的workspace資料夾,新建一個資料夾SbtWordCount,然後新建build.sbt檔案,輸入如下配置資訊

name := "sbt-wordcount" 
  
version := "1.0"

scalaVersion := "2.10.6"
autoScalaLibrary := false
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.Resource
EclipseKeys.createSrc := EclipseCreateSrc.Default + EclipseCreateSrc.ManagedClasses

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-mllib_2.10" % "1.5.2" % "provided",
  "org.apache.spark" % "spark-examples_2.10" % "1.1.1" % "provided"
)

resolvers ++= Seq( 
      // HTTPS is unavailable for Maven Central  
      "Maven Repository"     at "http://repo.maven.apache.org/maven2",  
      "Apache Repository"    at "https://repository.apache.org/content/repositories/releases",  
      "JBoss Repository"     at "https://repository.jboss.org/nexus/content/repositories/releases/",  
      "MQTT Repository"      at "https://repo.eclipse.org/content/repositories/paho-releases/",  
      "Cloudera Repository"  at "http://repository.cloudera.com/artifactory/cloudera-repos/",
      "le_bigdata_mining"    at "http://10.150.144.28/nexus/content/repositories/releases/",  
      Resolver.mavenLocal  
)
2、在命令列視窗中,進入到SbtWordCount目錄中,然後輸入“sbt eclipse”,等待出現Successfully created Eclipse ....就可以了

3、進入Scala IDE (eclipse)中,import該專案

4、若發現沒有src目錄,就手動建立。要想在windows下本地執行spark,還得需要winutils.exe,所以將其放在null/bin目錄中,可以參考上一篇文章windows中用scala-IDE開發spark—— WordCount