ol7.7安裝部署4節點spark3.0.0分散式叢集的詳細教程
阿新 • • 發佈:2020-07-12
為學習spark,虛擬機器中開4臺虛擬機器安裝spark3.0.0
底層hadoop叢集已經安裝好,見ol7.7安裝部署4節點hadoop 3.2.1分散式叢集學習環境
首先,去http://spark.apache.org/downloads.html下載對應安裝包
解壓
[hadoop@master ~]$ sudo tar -zxf spark-3.0.0-bin-without-hadoop.tgz -C /usr/local [hadoop@master ~]$ cd /usr/local [hadoop@master /usr/local]$ sudo mv ./spark-3.0.0-bin-without-hadoop/ spark [hadoop@master /usr/local]$ sudo chown -R hadoop: ./spark
四個節點都新增環境變數
export SPARK_HOME=/usr/local/spark export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
配置spark
spark目錄中的conf目錄下cp ./conf/spark-env.sh.template ./conf/spark-env.sh後面新增
export SPARK_MASTER_IP=192.168.168.11 export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop export SPARK_LOCAL_DIRS=/usr/local/hadoop export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
然後配置work節點,cp ./conf/slaves.template ./conf/slaves修改為
master
slave1
slave2
slave3
寫死JAVA_HOME,sbin/spark-config.sh最後新增
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_191
複製spark目錄到其他節點
sudo scp -r /usr/local/spark/ slave1:/usr/local/ sudo scp -r /usr/local/spark/ slave2:/usr/local/ sudo scp -r /usr/local/spark/ slave3:/usr/local/ sudo chown -R hadoop ./spark/
...
啟動叢集
先啟動hadoop叢集/usr/local/hadoop/sbin/start-all.sh
然後啟動spark叢集
通過master8080埠監控
完成安裝
到此這篇關於ol7.7安裝部署4節點spark3.0.0分散式叢集的詳細教程的文章就介紹到這了,更多相關ol7.7安裝部署spark叢集內容請搜尋我們以前的文章或繼續瀏覽下面的相關文章希望大家以後多多支援我們!