2. 程式人生 > >Sqoop的安裝和驗證

Sqoop的安裝和驗證

阿新 • • 發佈：2018-07-23
connector RKE bin 關系 user linu cos req config

　　Sqoop是一個用來完成Hadoop和關系型數據庫中的數據相互轉移的工具，它可以將關系型數據庫中的數據導入到Hadoop的HDFS中，也可以將HDFS的數據導入到關系型數據庫中。

　　Kafka是一個開源的分布式消息訂閱系統

　　一、Sqoop的安裝

　　1.http://www-eu.apache.org/dist/sqoop/1.4.7/下載sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz並解壓到/home/jun下

[jun@master sqoop-1.4.7.bin__hadoop-2.6.0]$ ls -l
total 2020
drwxr-xr-x. 2 jun jun    4096
 
 Dec 19  2017 bin
-rw-rw-r--. 1 jun jun   55089 Dec 19  2017 build.xml
-rw-rw-r--. 1 jun jun   47426 Dec 19  2017 CHANGELOG.txt
-rw-rw-r--. 1 jun jun    9880 Dec 19  2017 COMPILING.txt
drwxr-xr-x. 2 jun jun     150 Dec 19  2017 conf
drwxr-xr-x. 5 jun jun     169 Dec 19  2017 docs
drwxr-xr-x. 2 jun jun      96
 
 Dec 19  2017 ivy
-rw-rw-r--. 1 jun jun   11163 Dec 19  2017 ivy.xml
drwxr-xr-x. 2 jun jun    4096 Dec 19  2017 lib
-rw-rw-r--. 1 jun jun   15419 Dec 19  2017 LICENSE.txt
-rw-rw-r--. 1 jun jun     505 Dec 19  2017 NOTICE.txt
-rw-rw-r--. 1 jun jun   18772 Dec 19  2017 pom-old.xml
-rw-rw-r--. 1 jun jun    1096
 
 Dec 19  2017 README.txt
-rw-rw-r--. 1 jun jun 1108073 Dec 19  2017 sqoop-1.4.7.jar
-rw-rw-r--. 1 jun jun    6554 Dec 19  2017 sqoop-patch-review.py
-rw-rw-r--. 1 jun jun  765184 Dec 19  2017 sqoop-test-1.4.7.jar
drwxr-xr-x. 7 jun jun      73 Dec 19  2017 src
drwxr-xr-x. 4 jun jun     114 Dec 19  2017 testdata

　　2.配置MySQL連接器

[jun@master sqoop-1.4.7.bin__hadoop-2.6.0]$ cp /home/jun/Resources/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar   /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

　　3.配置Sqoop環境變量

　　編輯配置文件

[jun@master lib]$ cd /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/conf/
[jun@master conf]$ ls
oraoop-site-template.xml  sqoop-env-template.cmd  sqoop-env-template.sh  sqoop-site-template.xml  sqoop-site.xml
[jun@master conf]$ cp sqoop-env-template.sh sqoop-env.sh
[jun@master conf]$ gedit sqoop-env.sh

　　增加下面的配置

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/jun/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/jun/hadoop

#set the path to where bin/hbase is available
export HBASE_HOME=/home/jun/hbase-1.2.6.1

#Set the path to where bin/hive is available
export HIVE_HOME=/home/jun/apache-hive-2.3.3-bin

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zk

　　4.配置linux環境變量

#sqoop
export SQOOP_HOME=/home/jun/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin

　　5.啟動Sqoop，如果出現下面的內容就說明安裝成功

[jun@master ~]$ sqoop-help
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 15:56:36 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See ‘sqoop help COMMAND‘ for information on a specific command.

　　6.測試與MySQL的連接

　　（1）列出MySQL的所有數據庫

[jun@master ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306 --username root -P
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:03:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password: 
18/07/23 16:03:05 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:03:05 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive_db
mysql
performance_schema
sys

　　（2）列出數據庫下的所有數據表

[jun@master ~]$ sqoop-list-tables --connect jdbc:mysql://localhost:3306/mysql --username root -P
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:06:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password: 
18/07/23 16:06:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:06:09 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
columns_priv
db
engine_cost
event
func
general_log
gtid_executed
help_category
help_keyword
help_relation
help_topic
innodb_index_stats
innodb_table_stats
ndb_binlog_index
plugin
proc
procs_priv
proxies_priv
server_cost
servers
slave_master_info
slave_relay_log_info
slave_worker_info
slow_log
tables_priv
time_zone
time_zone_leap_second
time_zone_name
time_zone_transition
time_zone_transition_type
user

　　（3）執行MySQL的查詢語句

[jun@master ~]$ sqoop-eval --connect jdbc:mysql://localhost:3306/mysql --username root -P --query "select * from plugin"
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:09:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password: 
18/07/23 16:09:36 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:09:37 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
-----------------------------------------------
| name                 | dl                   | 
-----------------------------------------------
| validate_password    | validate_password.so | 
-----------------------------------------------

Sqoop的安裝和驗證

相關推薦

Sqoop的安裝和驗證

connector RKE bin 關系 user linu cos req config 　　Sqoop是一個用來完成Hadoop和關系型數據庫中的數據相互轉移的工具，它可以將關系型數據庫中的數據導入到Hadoop的HDFS中，也可以將HDFS的數據導入到關系型數據庫中。

druid叢集的安裝和驗證

核心 1、環境和架構 2、druid的安裝 3、druid的配置 4、overlord json 5、overlord csv 1、druid 環境和架構 環境資訊 Centos6.5 32GB 8C *5 Zookeeper 3.4.5

sqoop安裝和使用

如果不想寫專門的HDFS入庫程式，或者HIVE的入庫程式時，可以使用apache下sqoop工具 安裝比較簡單，從apache映象中下載下來sqoop解壓後即可使用，記得將驅動的jar包copy到sqoop/lib下，注意叢集必須是開啟的。 apache映象地址： ht

nginx安裝和測試 （已驗證）

fig figure lib pan 首頁 min 正常 如果 nginx安裝 進入：/usr/local/nginx 目錄註意：為了保證各插件之間的版本兼容和穩定，建議先通過以下版本進行測試驗證。一、下載版本 下載nginx: wget http://nginx.o

linux環境下pytesseract的安裝和央行征信中心的登錄驗證碼識別實戰

int tab 權限 linux a-z 都是 提示 解釋 text 首先是安裝，我參考的是這個 http://blog.csdn.net/xinghun_4/article/details/47860645 我是centos，使用yum yum install pyt

大數據學習之十五——sqoop的安裝和使用

dfs 9.1 pos 屬性 操作 所有 alt 成功 nbsp 1.概念了解 sqoop主要用於hadoop與傳統的數據庫（mysql、postgresql...）間進行數據的傳遞，可以將一個關系型數據庫（例如：MYSQL，Oracle，Postgrep等）中的數據導到h

sqoop(1):安裝和配置

一、功能 在大資料處理之後，需要資料展示，所以，要把相對較少的資料儲存到關係型資料庫中，然後做一套框架給外部查詢展示。sqoop的作用就是把資料從hdfs或者hive中匯出到mysql等關係型資料庫 二、sqoop部署 1.解壓  tar -zxvf sqoop-1.4.7

linux環境下jdk的安裝和配置 （已驗證）

一、軟體下載1、下載JDK（下面分別是32位系統和64位系統下的版本）http://download.oracle.com/otn-pub/java/jdk/7u9-b05/jdk-7u9-linux-i586.tar.gz?AuthParam=1350717577_c08

ISA2006安裝和部署基礎(虛擬機非域環境)

策略 ron 隧道 wan 對話 什麽 mark war fontsize 0x00. 為了測試基於HTTP隧道的繞過ISA,必須搭建模擬環境,為了不麻煩,我們這裏不配合域環境認證.本次實驗利用Vmware 10.0搭建好開發環境,實現ISA2006安裝和部署,同一時候

版本管理工具小烏龜TortoiseGit的安裝和使用(1)

姓名 save 使用 一行 https 技術 安裝路徑 管理工具 gen 1、軟件的安裝:1.1 安裝 Git使用軟件管理工具搜索 Git: 先安裝 Git(也可以在 https://git-for-windows.github.io/ 找到最新版),全部選擇默認即可;再安

樹莓派系統安裝和調試 總結整理篇

exp file 我們 log 令行 key 博客園 projects yun 第一次拿到樹莓派的時候，覺得它好小，就像一個小電路板一樣，經過對它的一番研究，感覺其實這個小電腦性能還是可以的，拿來運行一些小的程序、應用還是可以的，而且在有些情況下體積小就是它的優勢。閑話

mysql5.7：mysql安裝和基於SSL加密的主從復制（詳細剖析）

mysql ssl db 數據 加密傳輸 小生博客：http://xsboke.blog.51cto.com 小生 Q Q：1770058260 -------謝謝您的參考，如有疑問，歡迎交流目錄：--------my

工作流調度器azkaban的安裝和使用

用戶名 color smtp mail tex 服務器 重新 sts 建立 為什麽需要工作流調度系統 一個完整的數據分析系統通常都是由大量任務單元組成： 　　　　shell腳本程序，java程序，mapreduce程序、hive腳本等 各任務單元之間存在時間先後及前後

CloudStack+XenServer詳細部署方案 CloudStack管理節點的安裝和配置

cloudstack+xenserver詳細部署方案 cloudstack管理節點的安裝和配置CloudStack+XenServer詳細部署方案 CloudStack管理節點的安裝和配置本文將根據設計文檔, 安裝和配置CloudStack管理節點。本文只對配置流程和結果進行舉例說明, 具體 細節和配置操作請

Centos7下關於memcached的安裝和簡單使用

entos name 終端 說明 所有 local memcach 們的 clas 在這裏，由於用編譯安裝memcached服務端過於復雜，因此我選用依賴管理工具 yum 來實現 memcached 的服務端安裝： [root@localhost /]# yum insta

mycat安裝和測試

slave roo character 三臺 delet 1.4 property 啟動項 客戶端 mycat安裝和測試 一． 環境準備 本機環境是三臺centos6.5 IP 主機名 數據庫名 安裝軟件 192.168.17.4 mast

kafka安裝和使用

grep keys operator comment 隨機 寫入 實時流處理 生產者 keyword kafka安裝和啟動 kafka的背景知識已經講了很多了，讓我們現在開始實踐吧，假設你現在沒有Kafka和ZooKeeper環境。 Step 1: 下

Ubuntu 10.10安裝和使用pythonbrew簡介

添加 efi bz2 推薦 clean 刪除 pytho 裝包 undefine http://blog.csdn.net/nuccch/article/details/17595161 一、安裝Pythonbrew 通過pythonbrew工具實現多版本Python管理，

centos7 mysql數據庫安裝和配置

web 大小 images etc 安裝mysql 安裝 commands 0 rows type 一、系統環境 yum update升級以後的系統版本為 [[email protected]/* */ yl]# cat /etc/redhat-relea

Android 開發之Windows環境下Android Studio安裝和使用教程(圖文詳細步驟)

9.png 虛擬機 jdk版本 編寫 clip 開發平臺 集成開發 arc 電腦安裝 鑒於谷歌最新推出的Android Studio備受開發者的推崇，所以也跟著體驗一下。 一、介紹Android Studio Android Studio 是一個Android