1. 程式人生 > >sqoop的安裝 CentOS安裝mysql原始碼包

sqoop的安裝 CentOS安裝mysql原始碼包

Sqoop是一個用來完成Hadoop和關係型資料庫中的資料相互轉移的工具,

他可以將關係型資料庫(MySql,Oracle,Postgres等)中的資料匯入Hadoop的HDFS中,

也可以將HDFS的資料匯入關係型資料庫中。

前期準備:

1.Sqoop需要安裝在成功部署的Hadoop平臺上,並且要求Hadoop已經正常啟動。hadoop叢集的搭建

2.安裝了mysql資料庫,並且已經啟動.CentOS安裝mysql原始碼包

3.下載mysql-connector-java-5.1.12.tar.gz。

安裝過程;

1.下載sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz壓縮包。

2.上傳至/usr/local/src

3.解壓縮並且移動到上級目錄(/usr/local)

4.解壓縮mysql-connector-java-5.1.12.tar.gz檔案,

並且複製檔案mysql-connector-java-5.1.12-bin.jar到/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib中。

5.配置環境變數  

cd   /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/conf
cp sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh

根據情況新增配置資訊 內容如下:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the 
"License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # included in all the hadoop scripts with source command # should not be executable directly # also should not be passed any arguments, since we need original $* # Set Hadoop-specific environment variables here. #Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/usr/local/hadoop-2.6.5 #Set path to where hadoop-*-core.jar is available export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.6.5 #set the path to where bin/hbase is available export HBASE_HOME=/usr/local/hbase-1.4.8 #Set the path to where bin/hive is available export HIVE_HOME=/usr/local/apache-hive-2.1.0-bin #Set the path for where zookeper config dir is export ZOOCFGDIR=/usr/local/zookeeper/conf

6.配置linux的環境變數

cd
vim .bash_profile

在結尾新增如下程式碼:

#sqoop
export SQOOP_HOME=/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/
export PATH=$PATH:$SQOOP_HOME/bin

7.儲存退出;時檔案生效:

source .bash_profile

8.測試與mysql的連線:

cd   /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/bin
sqoop list-databases --connect jdbc:mysql://localhost:3306 --username root -P

輸入密碼後會顯示具體的資料庫列表

9.檢視具體資料庫中的 表資訊:

sqoop list-tables --connect jdbc:mysql://localhost:3306/db1 --username root -P

10.執行查詢語句:

sqoop -eval --connect jdbc:mysql://localhost:3306/db1 --username root  -P --query "select * from a limit 5"