通常會在IDE中編制程式，然後打成jar包，然後提交到叢集，最常用的是建立一個Maven專案，利用Maven來管理jar包的依賴。

一、生成WordCount的jar包

1. 開啟IDEA，File→New→Project→Maven→Next→填寫Groupld和Artifactld→Next→Finish

2. 配置Maven的pom.xml（配置好pom.xml以後，點選Enable Auto-Import即可）：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.wu</groupId>
    <artifactId>sparkWordCount</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.10.6</scala.version>
        <scala.compat.version>2.10</scala.compat.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.5.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>1.5.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.2</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-make:transitive</arg>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.wu.WordCount</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
 

注意：這裡需要修改Hadoop版本

3：將src/main/java和src/test/java分別修改成src/main/scala和src/test/scala，與pom.xml中的配置保持一致（）；

操作：java→Refactor→Rename

4：新建一個com.bie包，再新建一個scala class，型別為Object，spark程式如下：

package com.wu

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]): Unit = {
    //建立SparkConf()並且設定App的名稱
    val conf = new SparkConf().setAppName("wordCount");
    //建立SparkContext,該物件是提交spark app的入口
    val sc = new SparkContext(conf);
    //使用sc建立rdd,並且執行相應的transformation和action
    sc.textFile(args(0)).flatMap(_.split(" ")).map((_ ,1)).reduceByKey(_ + _,1).sortBy(_._2,false).saveAsTextFile(args(1));
    //停止sc，結束該任務
    sc.stop();
  }
}
 

5. 修改pom.xml中的mainClass，使其和自己的類路徑對應起來：

6. 使用Maven打包：點選IDEA右側的Maven Project選項，點選Lifecycle,選擇clean和package，然後點選Run Maven Build：

等待編譯完成，選擇編譯成功的jar包，target/sparkWordCount-1.0-SNAPSHOT.jar

二、執行

1. 開啟xshell，檔案→新建連線

新建好後輸入使用者名稱和密碼，建立連線。

2. 使用Xftp新建檔案傳輸（Ctrl+Alt+F），將剛剛生成的jar包和WordCount拖拽至 /home/hdfs目錄下

3. 使用Xshell將WordCount.txt上傳至hdfs系統

切換至hdfs使用者：[[email protected] ~]# su hdfs

到spark的bin目錄下：[[email protected] root]$ cd /home/hdfs/software/spark/bin

在hdfs系統中新建input資料夾：hadoop fs -mkdir /input

檢視是否新建成功：[[email protected] bin]$ cd /home/hdfs/software/hadoop/bin     #轉到該目錄下

                                [[email protected] bin]$ ./hadoop fs -ls /

將txt檔案上傳至input資料夾：[[email protected] root]$ cd /home/hdfs/software/spark/bin      #轉回到該目錄

                                               [[email protected] bin]$ hadoop fs -put /home/hdfs/WordCount.txt /input

檢視是否上傳成功：[[email protected] bin]$ cd /home/hdfs/software/hadoop/bin     #轉到該目錄下

                                [[email protected] bin]$ ./hadoop fs -ls /input

返回hdfs使用者根目錄：cd ~

使用spark-submit命令提交Spark應用：[[email protected] ~]$ /home/hdfs/software/spark/bin/spark-submit --class com.bie.WordCount sparkWordCount-1.0-SNAPSHOT.jar hdfs://data2.cshdp.com:9000/input/WordCount.txt hdfs://data2.cshdp.com:9000/output

檢視執行結果：[[email protected] bin]$ cd /home/hdfs/software/hadoop/bin     #轉到該目錄下

                         [[email protected] bin]$ ./hadoop fs -ls /output

                         [[email protected] bin]$ ./hadoop fs -cat /output/part-00000   #檢視檔案內容