1， 配置JAVA_HOME

通過yum等方式，可以安裝最新的java版本（yum install java）。安裝完java後，我們需要配置JAVA_HOME，首先要確定jdk安裝的位置：

[[email protected] marshall]# which java 

/usr/bin/java

[[email protected] marshall]#  

[[email protected] marshall]# ls -lrt/usr/bin/java 

lrwxrwxrwx. 1 root root 22 Feb 11 19:33/usr/bin/java -> /etc/alternatives/java

[[email protected] marshall]#  

[[email protected] marshall]# ls -lrt/etc/alternatives/java

lrwxrwxrwx. 1 root root 73 Feb 11 19:33/etc/alternatives/java ->/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/bin/java

然後編輯/etc/profile 檔案，加入：

export SCALA_HOME=/opt/scala/scala-2.12.4

exportSPARK_HOME=/opt/spark-2.2.1-bin-hadoop2.7

exportJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64

export JRE_HOME=$JAVA_HOME/jre

exportCLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

PATH=$PATH:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$JAVA_HOME/bin:$JRE_HOME/bin

export PATH

然後執行下面命令，配置生效。

source /etc/profile

驗證配置：

[[email protected] opt]# java -version

openjdk version "1.8.0_161"

OpenJDK Runtime Environment (build1.8.0_161-b14)

OpenJDK 64-Bit Server VM (build 25.161-b14,mixed mode)

2 安裝scala

執行命令解壓縮，這裡，我將安裝檔案都放入了/opt/檔案下，執行解壓縮命令：

tar –xvf scala-2.12.4

配置見java_home章節

驗證配置：

[[email protected] opt]# scala -version

Scala code runner version 2.12.4 --Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.

3 安裝spark

然後解壓縮到/opt/目錄。Profile配置見java_home部分，切換到spark安裝目錄的conf目錄下：

cp –rp  slaves.template slaves

cp –rp  spark-env.sh.template spark-env.sh

編輯slaves：

localhost

編輯 spark-env.sh

export SCALA_HOME=/opt/scala/scala-2.12.4

exportSPARK_HOME=/opt/spark-2.2.1-bin-hadoop2.7

exportJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64

export SPARK_MASTER_IP=hserver1

export SPARK_EXECUTOR_MEMORY=1G

測試：

cd /opt/spark-2.2.1-bin-hadoop2.7

./bin/run-example   SparkPi  10

編譯完成後，輸入命令：

[[email protected] spark-2.2.1-bin-hadoop2.7]#./bin/spark-shell

Using Spark's default log4j profile:org/apache/spark/log4j-defaults.properties

Setting default log level to"WARN".

To adjust logging level usesc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

18/02/21 19:05:27 WARN NativeCodeLoader:Unable to load native-hadoop library for your platform... using builtin-javaclasses where applicable

Spark context Web UI available athttp://192.168.44.152:4040

Spark context available as 'sc' (master =local[*], app id = local-1519268738326).

Spark session available as 'spark'.

Welcome to

     ____              __

    / __/__  ___ _____/ /__

   _\ \/ _ \/ _ `/ __/  '_/

  /___/ .__/\_,_/_/ /_/\_\   version2.2.1

     /_/

Using Scala version 2.11.8 (OpenJDK 64-BitServer VM, Java 1.8.0_161)

Type in expressions to have them evaluated.

Type :help for more information.

scala>