因為是從零開始，所以本文將spark部署成單機模式，並且有些檔案放到home的個人目錄中，不過看下設定的環境變數的就可以知道，這些檔案放那裡其實是無所謂的

伺服器環境為cenos，並且JDK已經正確安裝，可通過jar命令是否可用來判斷

$ jar
Usage: jar {ctxui}[vfmn0PMe] [jar-file] [manifest-file] [entry-point] [-C dir] files ...
Options:
    -c  create new archive
    -t  list table of contents for archive
...

SCALA安裝

wget http://www.scala-lang.org/files/archive/scala-2.10.1.tgz
tar zxf scala-2.10.1.tgz
sudo mv scala-2.10.1 /usr/local

在/etc/profile.d/下建立檔案scala.sh，內容如下

$ cat /etc/profile.d/scala.sh 
#!usr/bin/env bash
export SCALA_HOME=/usr/local/scala-2.10.1
export PATH=$PATH:$SCALA_HOME/bin

使環境變數生效

cd /etc
source profile

SBT安裝

wget https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar
$cd ~/
$mkdir bin --如果已經存在這個目錄則不需要重複建立
$mv sbt-launch.jar ~/bin

在~/bin目錄下建立檔案sbt，內容如下

$ cat sbt
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
java $SBT_OPTS -jar `dirname $0`
 
/sbt-launch.jar "[email protected]"

按如下方式測試，看是否OK

$ sbt
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[info] Set current project to liujinliu (in build file:/home/liujinliu/)

安裝SPARK

wget http://mirrors.hust.edu.cn/apache/spark/spark-1.3.1/spark-1.3.1-bin-hadoop2.6.tgz
tar zxf spark-1.3.1-bin-hadoop2.6.tgz

啟動SPARK

./sbin/start-master.sh

登入localhost:8080可以看到UI介面，上面會顯示當前節點的資訊
其中在標題處會顯示當前master所在的URL資訊，這個引數需要作為入參在建立worker時候使用(spark://IP:PORT)
啟動一個worker

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT

此時localhost:8080的頁面可以看到worker的資訊

載入之後解壓，目錄結構如下

base/
    build.sbt
    README.md
    src/
        main/
            scala/
                 AccessLogParser.scala
                 AccessLogRecord.scala
                 SimpleTestApp.scala --這個檔案為自己建立

在src下面有一個和main同級的目錄test，刪除，下面將講解如何將這個工程編譯打包然後提交給spark執行
SimpleTestApp.scala內容如下

package com.alvinalexander.accesslogparser

import java.util.Calendar

object SimpleTestApp {
    def main(args: Array[String]) {
        val records = """
124.30.9.161 - - [21/Jul/2009:02:48:11 -0700] "GET /java/edu/pj/pj010004/pj010004.shtml HTTP/1.1" 200 16731 "http://www.google.co.in/search?hl=en&client=firefox-a&rlz=1R1GGGL_en___IN337&hs=F0W&q=reading+data+from+file+in+java&btnG=Search&meta=&aq=0&oq=reading+data+" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 GTB5"
89.166.165.223 - - [21/Jul/2009:02:48:12 -0700] "GET /favicon.ico HTTP/1.1" 404 970 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11"
94.102.63.11 - - [21/Jul/2009:02:48:13 -0700] "GET / HTTP/1.1" 200 18209 "http://www.developer.com/net/vb/article.php/3683331" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
124.30.7.162 - - [21/Jul/2009:02:48:13 -0700] "GET /images/tline3.gif HTTP/1.1" 200 79 "http://www.devdaily.com/java/edu/pj/pj010004/pj010004.shtml" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 GTB5"
122.165.54.17 - - [21/Jul/2009:02:48:12 -0700] "GET /java/java_oo/ HTTP/1.1" 200 32579 "http://www.google.co.in/search?hl=en&q=OO+with+java+standalone+example&btnG=Search&meta=&aq=f&oq=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
217.32.108.226 - - [21/Jul/2009:02:48:13 -0700] "GET /blog/post/perl/checking-testing-perl-module-in-inc-include-path/ HTTP/1.1" 200 18417 "http://www.devdaily.com/blog/post/perl/perl-error-cant-locate-module-in-inc/" "Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7) Gecko/20070606"
122.165.54.17 - - [21/Jul/2009:02:48:15 -0700] "GET /java/java_oo/java_oo.css HTTP/1.1" 200 1235 "http://www.devdaily.com/java/java_oo/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
""".split("\n").filter(_ != "")
        val parser = new AccessLogParser
        val rec = parser.parseRecord(records(0))
        println("IP ADDRESS: " + rec.get.clientIpAddress)
    }
}

這個檔案的測試內容是參考原始碼test目錄的兩個檔案寫的

修改build.sbt內容:

name := "ScalaApacheAccessLogParser"
version := "1.0"
scalaVersion := "2.10.4"  --主要是這行，與當前使用的scala版本匹配
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
scalacOptions += "-deprecation"
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.1.0" % "test"

在base目錄下執行編譯和打包

sbt  compile
sbt  package

需要注意的是sbt 0.13.x版本一定要和scala 2.10.x版本配套使用，使用scala 2.11.x版本會報錯(can’t expand macros compiled by previous versions of Scala)

編譯成功會在base目錄下生成target資料夾，並且可以在這個資料夾的子資料夾內找到scalaapacheaccesslogparser_2.10-1.0.jar，路徑為
target/scala-2.10/scalaapacheaccesslogparser_2.10-1.0.jar

將target資料夾整體拷貝到與
spark-1.3.1-bin-hadoop2.6資料夾平行的位置

spark/
      spark-1.3.1-bin-hadoop2.6/
      target/

提交driver到spark

./spark-1.3.1-bin-hadoop2.6/bin/spark-submit --class "com.alvinalexander.accesslogparser.SimpleTestApp" --master spark://IP:PORT target/scala-2.10/scalaapacheaccesslogparser_2.10-1.0.jar

IP和PORT跟起worker時候的一樣
可以在shell看到如下輸出

IP ADDRESS: 124.30.9.161
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

另外安利我寫的兩個模組，歡迎使用或貢獻程式碼：