下面簡單記錄mac單機spark安裝測試的過程

• 單機Mac安裝spark並做簡單yarn模式shell測試
• 配置SparkSQL查詢Hive
• spark-sql CLI 查詢Hive

一、安裝Spark

1-下載安裝scala

移動至/Users/使用者名稱xxx/Software/scala-2.10.4下

tar -zxvf scala-2.10.4.tgz

配置環境變數

vi ~/.bash_profile

export SCALA_HOME=/Users/使用者名稱xxx/Software/scala-2.10.4

export PATH=PATH:PATH:{SCALA_HOME}/bin

source
 
 

測試

scala -version

2-下載安裝Spark

http://spark.apache.org/downloads.html
下載spark-2.2.1-bin-hadoop2.7.tgz.
解壓至/Users/使用者名稱xxx/Documents/software/spark
spark-2.2.1 資料夾及子檔案的和使用者設為hadoop的安裝使用者如使用者名稱xxx

chown -R 使用者名稱xxx spark-2.2.1

修改conf資料夾spark-env.sh和slaves

cp spark-env.sh.template spark-env.sh

在spark-env.sh新增

export SCALA_HOME=/Users/使用者名稱xxx/Software/scala-2.10.4
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.121-2.6.8.0.el7_3.x86_64
export SPARK_MASTER_IP=localhost
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/data/hadoop-2.7.3/etc/hadoop

具體可以執行下面的命令

chown -R 使用者名稱比如使用者名稱xxx spark2.2.1
 

cp spark-env.sh.template spark-env.sh
echo "export SCALA_HOME=/Users/使用者名稱xxx/Software/scala-2.10.4" >> spark-env.sh
echo "export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/home" >> spark-env.sh
echo "export SPARK_MASTER_IP=localhost" >> spark-env.sh
echo "export SPARK_WORKER_MEMORY=1g" >> spark-env.sh
echo "export HADOOP_CONF_DIR=/Users/使用者名稱xxx/Documents/software/hadoop/hadoop-2.7.4/etc/hadoop" >> spark-env.sh

這裡單機不用配置slaves

二、啟動Spark並做簡單yarn模式的shell測試

1-啟動Spark

在sbin資料夾下執行， start-all.sh

localhost:~/Documents/software/spark/spark2.2.1/sbin$ jps. 
39793 Master
39826 Worker

2-WebUI

hive> load data local inpath '/Users/hjw/Documents/software/hive/student.txt' into table hive_test.student;
Loading data to table hive_test.student
OK
Time taken: 0.386 seconds
查詢資料
hive> select * from hive_test.student;
OK
1001 zhangsan
1002 lisi
Time taken: 1.068 seconds, Fetched: 2 row(s)

3-啟動yarn mode的spark-shell

yarn模式啟動spark-shell

spark2.2.1/bin$ spark-shell –num-executors 1 –executor-memory 512m –master yarn-client

4-執行測試

在spark-shell中執行

val file=sc.textFile("hdfs://localhost:9000/user/hive/warehouse/hive_test.db/student")
val rdd = file.flatMap(line => line.split("\t")).map(word => (word,1)).reduceByKey(_+_)
rdd.take(10)

scala> val file=sc.textFile("hdfs://localhost:9000/user/hive/warehouse/hive_test.db/student")
file: org.apache.spark.rdd.RDD[String] = hdfs://localhost:9000/user/hive/warehouse/hive_test.db/student MapPartitionsRDD[39] at textFile at <console>:24

scala> val rdd = file.flatMap(line => line.split("\t")).map(word => (word,1)).reduceByKey(_+_)
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[42] at reduceByKey at <console>:26

scala> rdd.take(10)
res12: Array[(String, Int)] = Array((zhangsan,1), (1001,1), (1002,1), (lisi,1))

5-退出spark-shell

退出spark-shell,輸入 “:quit”

6-關閉spark

/software/spark/spark2.2.1/sbin$ ./stop-all.sh
localhost: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master

三、配置SparkSQL讀取Hive資料

為了讓Spark能夠連線到Hive資料倉庫，需要將Hive中的hive-site.xml檔案拷貝到Spark的conf目錄下，Spark通過這個配置檔案找到Hive的元資料，完成資料讀取。

1-已安裝單機Hive

hive –service metastore &

啟動hive shell查詢資料

hive> select * from student;
OK
1001 zhangsan
1002 lisi
Time taken: 1.164 seconds, Fetched: 2 row(s)

2-將Hive中hive-site.xml檔案拷貝到Spark的conf目錄

• 將Hive中的hive-site.xml檔案拷貝到Spark的conf目錄
• 重新啟動spark，在sbin資料夾下 執行：start-all.sh

3-啟動spark-shell

yarn模式啟動spark-shell，spark2.2.1/bin$ spark-shell –num-executors 1 –executor-memory 512m –master yarn-client

4-SparkSQL查詢Hive資料

scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
sqlContext: org.apache.spark.sql.hive.HiveContext = [email protected]27210a3b

scala> sqlContext.sql(" SELECT * FROM hive_test.student")
18/01/11 22:15:00 WARN conf.HiveConf: HiveConf of name hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist
18/01/11 22:15:00 WARN conf.HiveConf: HiveConf of name hive.llap.io.orc.time.counters does not exist
18/01/11 22:15:00 WARN conf.HiveConf: HiveConf of name hive.tez.task.scale.memory.reserve-fraction.min does not exist
18/01/11 22:15:00 WARN conf.HiveConf: HiveConf of name hive.orc.splits.ms.footer.cache.ppd.enabled does not exist
18/01/11 22:15:00 WARN conf.HiveConf: HiveConf of name hive.server2.metrics.enabled does not exist

。。。。。

18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.msck.path.validation does not exist
18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.tez.task.scale.memory.reserve.fraction does not exist
18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.compactor.history.reaper.interval does not exist
18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.tez.input.generate.consistent.splits does not exist
18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.server2.xsrf.filter.enabled does not exist
18/01/11 22:15:01 WARN conf.HiveConf: HiveConf of name hive.llap.io.allocator.alloc.max does not exist
res0: org.apache.spark.sql.DataFrame = [id: int, name: string]

以上有錯誤，報錯沒有啟動hive server，另開一個終端執行：hive –service hiveserver2 &，啟動hiveserver2

t:~$ hive --service hiveserver2 &
[1] 43667

spark-shell，重新查詢Hive

scala> sqlContext.sql(" SELECT * FROM hive_test.student")
res1: org.apache.spark.sql.DataFrame = [id: int, name: string]

scala> sqlContext.sql(" SELECT * FROM hive_test.student").collect().foreach(println)
[1001,zhangsan]
[1002,lisi]

四、spark-sql CLI 查詢

將hive-site.xml複製到spark的conf下,執行
下載mysql-connector-java-5.1.45-bin.jar，
https://dev.mysql.com/downloads/file/?id=474257
放在了/Users/xxx/Documents/software/spark/spark2.2.1/classpath

.spark2.2.1/bin$ spark-sql --driver-class-path /Users/xxx/Documents/software/spark/spark2.2.1/classpath/mysql-connector-java-5.1.45-bin.jar

嘗試新增--driver-class-path 也可以，比如直接執行spark-sql，或者spark-sql –master yarn

在啟動spark-sql時，如果不指定master，則以local的方式執行，master既可以指定standalone的地址，也可以指定yarn；
當設定master為yarn時(spark-sql –master yarn)時，可以通過http://hadoop000:8088頁面監控到整個job的執行過程；
注：如果在$SPARK_HOME/conf/spark-defaults.conf中配置了spark.master spark://hadoop000:7077，那麼在啟動spark-sql時不指定master也是執行在standalone叢集之上。

進入spark-sql

spark-sql> select * from hive_test.student;
。。。
1001    zhangsan
1002    lisi
Time taken: 3.306 seconds, Fetched 2 row(s)

開始一直報錯，多少嘗試好了，原因再找吧，抓緊睡覺

開始報can not access hivemeta 
後來多次嘗試可以了，什麼原因？？？
INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"