java+scala+zookpeeper+flume+kafka+maven+hadoop+hbase+hive+spark安裝流程
阿新 • • 發佈：2019-01-01
系統版本使用的是centos7.0
開始安裝之前需要先配置下hosts
vi /etc/hosts
自己的IP hadoop000
關閉防火牆 systemctl stop firewalld.service
java使用的是1.8
java下載地址 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
haddop使用的版本都是cdh5.7.0版本
cdh5地址http://archive.cloudera.com/cdh5/cdh/5/
下面是我整理出來的下載地址,可以直接使用
scala https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz
zookpeeper http://archive.cloudera.com/cdh5/cdh/5/zookeeper-3.4.5-cdh5.7.0.tar.gz
flume http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.7.0.tar.gz
kafka https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
maven https://mirrors.tuna.tsinghua.edu.cn/apache//maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
hadoop http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
hbase http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz
hive http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.tar.gz
spark http://mirror.bit.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0.tgz
安裝包存放在/root/tools下，解壓後的應用存放在/root/app
一、安裝java
cd /root/tools
解壓 tar -zxvf jdk-8u151-linux-x64.tar.gz -C /root/app
將java配置系統環境變數中: ~/.bash_profile
export JAVA_HOME=/root/app/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: java -version
二、安裝scala
解壓 tar -zxvf scala-2.11.8.tgz -C /root/app/
將scala配置系統環境變數中: ~/.bash_profile
export SCALA_HOME=/root/app/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: scala -version
三、安裝zookeeper
解壓 tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C /root/app/
將zookeeper配置系統環境變數中: ~/.bash_profile
export ZOOKEEPER_HOME=/root/app/zookeeper-3.4.5-cdh5.7.0
export PATH=$ZOOKEEPER_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
修改zookeeper配置檔案
cd /root/app/zookeeper-3.4.5-cdh5.7.0/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
修改zookeeper的檔案儲存地址
dataDir=/root/tmp/zookeeper
mkdir -p /root/tmp/zookeeper
啟動zookeeper zkServer.sh start
jps 檢測是否啟動成功。出現QuorumPeerMain證明啟動成功
四、安裝flume
cd /root/tools
解壓 tar -zxvf flume-ng-1.6.0-cdh5.7.0.tar.gz -C /root/app/
將flume配置系統環境變數中: ~/.bash_profile
export FLUME_HOME=/root/app/apache-flume-1.6.0-cdh5.7.0-bin
export PATH=$FLUME_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: flume-ng version
五、安裝kafka
解壓 tar -zxvf kafka_2.11-0.9.0.0.tgz -C /root/app/
將kafka配置系統環境變數中: ~/.bash_profile
export KAFKA_HOME=/root/app/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
cd /root/app/kafka_2.11-0.9.0.0/config
修改配置檔案:
vi server.properties
host.name=hadoop000
log.dirs=/root/tmp/kafka-logs
zookeeper.connect=hadoop000:2181
mkdir -p /root/tmp/kafka-logs
執行kafka-server-start.sh啟動kafka
命令說明 kafka-server-start.sh [-daemon] server.properties [--override property=value]
啟動kafka kafka-server-start.sh $KAFKA_HOME/config/server.properties
六、安裝maven
解壓 tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /root/app/
將maven配置系統環境變數中: ~/.bash_profile
export MANVEN_HOME=/root/app/apache-maven-3.3.9
export PATH=$MANVEN_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確：mvn -version
七、安裝hadoop
配置ssh無密碼登入
ssh-keygen -t rsa
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
解壓tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /root/app/
將hadoop配置系統環境變數中: ~/.bash_profile
export HADOOP_HOME=/root/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
配置hadoop配置檔案
進入hadoop目錄 cd /root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
vi hadoop_env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
建立hadoop臨時檔案
mkdir /root/tmp/hadoop
vi core-site.xml
把下面的內容放到<configuration></configuration>中
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop000:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/tmp/hadoop</value>
</property>
vi hdfs-site.xml
修改hdfs的副本系數
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
格式化hdfs
hdfs namenode -format
進入hadoop根目錄
cd /root/app/hadoop-2.6.0-cdh5.7.0
啟動hdfs
sbin/start-dfs.sh
檢測hdfs是否啟動成功
jps 出現NameNode和DataNode證明啟動成功
配置yarn
cd etc/hadoop
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
vi yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
啟動yarn
cd ../../
sbin/start-yarn.sh
檢測yarn是否啟動成功
jps 出現NodeManager和ResourceManager證明啟動成功
八、安裝hbase
cd /root/tools
解壓tar -zxvf hbase-1.2.0-cdh5.7.0.tar.gz -C /root/app/
將hbase配置系統環境變數中: ~/.bash_profile
export HBASE_HOME=/root/app/hbase-1.2.0-cdh5.7.0
export PATH=$HBASE_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
修改hbase配置檔案
vi $HBASE_HOME/conf/hbase-env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
export HBASE_MANAGES_ZK=false
vi $HBASE_HOME/conf/hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop001:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hadoop001:2181</value>
</property>
啟動hbase
start-hbase.sh
驗證hbase啟動是否成功
jps 出現HMaster和HRegionServer證明啟動成功
九、安裝hive
解壓tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/app/
將hive配置系統環境變數中: ~/.bash_profile
export HIVE_HOME=/root/app/hive-1.1.0-cdh5.7.0
export PATH=$HIVE_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
十、安裝spark
解壓tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/source
cd /root/source/spark-2.2.0
編譯 ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
編譯報錯：failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.2.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in central (https://repo1.maven.org/maven2) -> [Help 1]
解決辦法: 在pom.xml新增:
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
編譯了一天一夜都沒有好,無奈,直接下載官方編譯好的版本吧!
cd /root/tools
下載編譯好的版本:wget http://mirrors.hust.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.6.tgz
解壓tar -zxvf spark-2.2.0-bin-hadoop2.6.tgz -C /root/app
將spark配置系統環境變數中: ~/.bash_profile
export SPARK_HOME=/root/app/spark-2.2.0-bin-hadoop2.6
export PATH=$SPARK_HOME/bin:$PATH
開始安裝之前需要先配置下hosts
vi /etc/hosts
自己的IP hadoop000
關閉防火牆 systemctl stop firewalld.service
java使用的是1.8
java下載地址 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
haddop使用的版本都是cdh5.7.0版本
cdh5地址http://archive.cloudera.com/cdh5/cdh/5/
下面是我整理出來的下載地址,可以直接使用
scala https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz
zookpeeper http://archive.cloudera.com/cdh5/cdh/5/zookeeper-3.4.5-cdh5.7.0.tar.gz
flume http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.7.0.tar.gz
kafka https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
maven https://mirrors.tuna.tsinghua.edu.cn/apache//maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
hadoop http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
hbase http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz
hive http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.tar.gz
spark http://mirror.bit.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0.tgz
安裝包存放在/root/tools下，解壓後的應用存放在/root/app
一、安裝java
cd /root/tools
解壓 tar -zxvf jdk-8u151-linux-x64.tar.gz -C /root/app
將java配置系統環境變數中: ~/.bash_profile
export JAVA_HOME=/root/app/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: java -version
二、安裝scala
解壓 tar -zxvf scala-2.11.8.tgz -C /root/app/
將scala配置系統環境變數中: ~/.bash_profile
export SCALA_HOME=/root/app/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: scala -version
三、安裝zookeeper
解壓 tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C /root/app/
將zookeeper配置系統環境變數中: ~/.bash_profile
export ZOOKEEPER_HOME=/root/app/zookeeper-3.4.5-cdh5.7.0
export PATH=$ZOOKEEPER_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
修改zookeeper配置檔案
cd /root/app/zookeeper-3.4.5-cdh5.7.0/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
修改zookeeper的檔案儲存地址
dataDir=/root/tmp/zookeeper
mkdir -p /root/tmp/zookeeper
啟動zookeeper zkServer.sh start
jps 檢測是否啟動成功。出現QuorumPeerMain證明啟動成功
四、安裝flume
cd /root/tools
解壓 tar -zxvf flume-ng-1.6.0-cdh5.7.0.tar.gz -C /root/app/
將flume配置系統環境變數中: ~/.bash_profile
export FLUME_HOME=/root/app/apache-flume-1.6.0-cdh5.7.0-bin
export PATH=$FLUME_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確: flume-ng version
五、安裝kafka
解壓 tar -zxvf kafka_2.11-0.9.0.0.tgz -C /root/app/
將kafka配置系統環境變數中: ~/.bash_profile
export KAFKA_HOME=/root/app/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
cd /root/app/kafka_2.11-0.9.0.0/config
修改配置檔案:
vi server.properties
host.name=hadoop000
log.dirs=/root/tmp/kafka-logs
zookeeper.connect=hadoop000:2181
mkdir -p /root/tmp/kafka-logs
執行kafka-server-start.sh啟動kafka
命令說明 kafka-server-start.sh [-daemon] server.properties [--override property=value]
啟動kafka kafka-server-start.sh $KAFKA_HOME/config/server.properties
六、安裝maven
解壓 tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /root/app/
將maven配置系統環境變數中: ~/.bash_profile
export MANVEN_HOME=/root/app/apache-maven-3.3.9
export PATH=$MANVEN_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
檢測安裝是否正確：mvn -version
七、安裝hadoop
配置ssh無密碼登入
ssh-keygen -t rsa
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
解壓tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /root/app/
將hadoop配置系統環境變數中: ~/.bash_profile
export HADOOP_HOME=/root/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
配置hadoop配置檔案
進入hadoop目錄 cd /root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
vi hadoop_env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
建立hadoop臨時檔案
mkdir /root/tmp/hadoop
vi core-site.xml
把下面的內容放到<configuration></configuration>中
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop000:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/tmp/hadoop</value>
</property>
vi hdfs-site.xml
修改hdfs的副本系數
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
格式化hdfs
hdfs namenode -format
進入hadoop根目錄
cd /root/app/hadoop-2.6.0-cdh5.7.0
啟動hdfs
sbin/start-dfs.sh
檢測hdfs是否啟動成功
jps 出現NameNode和DataNode證明啟動成功
配置yarn
cd etc/hadoop
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
vi yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
啟動yarn
cd ../../
sbin/start-yarn.sh
檢測yarn是否啟動成功
jps 出現NodeManager和ResourceManager證明啟動成功
八、安裝hbase
cd /root/tools
解壓tar -zxvf hbase-1.2.0-cdh5.7.0.tar.gz -C /root/app/
將hbase配置系統環境變數中: ~/.bash_profile
export HBASE_HOME=/root/app/hbase-1.2.0-cdh5.7.0
export PATH=$HBASE_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
修改hbase配置檔案
vi $HBASE_HOME/conf/hbase-env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
export HBASE_MANAGES_ZK=false
vi $HBASE_HOME/conf/hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop001:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hadoop001:2181</value>
</property>
啟動hbase
start-hbase.sh
驗證hbase啟動是否成功
jps 出現HMaster和HRegionServer證明啟動成功
九、安裝hive
解壓tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/app/
將hive配置系統環境變數中: ~/.bash_profile
export HIVE_HOME=/root/app/hive-1.1.0-cdh5.7.0
export PATH=$HIVE_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效
十、安裝spark
解壓tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/source
cd /root/source/spark-2.2.0
編譯 ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
編譯報錯：failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.2.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in central (https://repo1.maven.org/maven2) -> [Help 1]
解決辦法: 在pom.xml新增:
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
編譯了一天一夜都沒有好,無奈,直接下載官方編譯好的版本吧!
cd /root/tools
下載編譯好的版本:wget http://mirrors.hust.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.6.tgz
解壓tar -zxvf spark-2.2.0-bin-hadoop2.6.tgz -C /root/app
將spark配置系統環境變數中: ~/.bash_profile
export SPARK_HOME=/root/app/spark-2.2.0-bin-hadoop2.6
export PATH=$SPARK_HOME/bin:$PATH
source ~/.bash_profile 讓其配置生效