2。解壓就可以使用

3。建立一個scala專案，把所有spark的jar包都拷貝進去

4。然後書寫程式碼

import org.apache.spark._  
import SparkContext._  
import org.apache.spark.SparkContext  
import org.apache.spark.sql.SparkSession

object WordCount {  

   def main(args:Array[String]){
     print("...")

    val spark = SparkSession.builder
 
.appName("Simple Application").master("local").getOrCreate()
     val textFile = spark.read.textFile("E:/temp/data100000000-200000000-2000000000/crime_man1.txt")  
  /*  val result = textFile.flatMap(line => line.split(" "))  
        .map(word => (word, 1)).reduceByKey(_ + _) */ 
     print(textFile.count)
    spark.stop()
   }
}  
 

5.然後發現問題 unknown scala version problem

7。解決spark2.1.1使用scala2.11.8編譯的，將scalaSDK換成2.11.8問題得到解決

8.但是報錯java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.，這個錯誤只要不連線hdfs上的檔案進行測試，就沒事

下載hadoo2.7.3的Hadoop.dll和winutils.exe 這兩個檔案，把hadoop.dll放到本地電腦的C:\windows/system32下，重啟電腦，然後把winutils.exe檔案放到伺服器的/opt/hzjs/hadoop-2.7.3/bin/目錄下，每個叢集的機器都要放，把叢集的hadoop拷貝一份放到本地E:\02-hadoop\hadoop-2.7.3\目錄下，把裡面的配置檔案拷貝到專案上圖中。

沒有做這個winutils.exe會出現這樣的問題

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)

沒有做這個hadoop.dll會出現這樣的問題
Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z