Sqoop匯入MySQL所有表到Hive
阿新 • • 發佈：2019-02-12
1、匯入到Hive預設資料庫
[[email protected] sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18 hive> show tables;
OK
files
logs
t1
Time taken: 0.064 seconds, Fetched: 5 row(s)
hive> select count(*) from files;
Query ID = root_20180524152738_798fe3b4-acf1-48a1-a90a-5060fe30269a
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1526097883376_0017, Tracking URL = http://ndoe1:8088/proxy/application_1526097883376_0017/
Kill Command = /opt/hadoop-2.7.5/bin/hadoop job -kill job_1526097883376_0017
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2018-05-24 15:27:45,426 Stage-1 map = 0%, reduce = 0%
2018-05-24 15:27:51,784 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 5.4 sec
2018-05-24 15:27:59,017 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 8.39 sec
MapReduce Total cumulative CPU time: 8 seconds 390 msec
Ended Job = job_1526097883376_0017
MapReduce Jobs Launched:
Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 8.39 sec HDFS Read: 3816408 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 8 seconds 390 msec
OK
3244
Time taken: 21.453 seconds, Fetched: 1 row(s)
hive>
2、匯入到指定Hive資料庫
預設是匯入到default資料庫中，如果想指定匯入到某個資料庫中，可以使用–hive-database引數
hive> create database test;
OK
Time taken: 0.158 seconds
hive>[[email protected] sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --hive-database test --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18/05/24 16:00:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/05/24 16:00:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/05/24 16:00:06 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/05/24 16:00:06 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/05/24 16:00:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/05/24 16:00:06 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 16:00:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.jar
18/05/24 16:00:08 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/05/24 16:00:08 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/05/24 16:00:08 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/05/24 16:00:08 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/05/24 16:00:08 INFO mapreduce.ImportJobBase: Beginning import of files
18/05/24 16:00:08 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/05/24 16:00:09 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/05/24 16:00:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 16:00:12 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 16:00:12 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `files`
18/05/24 16:00:12 INFO db.IntegerSplitter: Split size: 810; Num splits: 4 from: 1 to: 3244
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: number of splits:4
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0019
18/05/24 16:00:12 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0019
18/05/24 16:00:12 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0019/
18/05/24 16:00:12 INFO mapreduce.Job: Running job: job_1526097883376_0019
18/05/24 16:00:20 INFO mapreduce.Job: Job job_1526097883376_0019 running in uber mode : false
18/05/24 16:00:20 INFO mapreduce.Job: map 0% reduce 0%
18/05/24 16:00:26 INFO mapreduce.Job: map 25% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job: map 100% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job: Job job_1526097883376_0019 completed successfully
18/05/24 16:00:28 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=570260
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=412
HDFS: Number of bytes written=3799556
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Killed map tasks=1
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=17383
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=17383
Total vcore-milliseconds taken by all map tasks=17383
Total megabyte-milliseconds taken by all map tasks=17800192
Map-Reduce Framework
Map input records=3244
Map output records=3244
Input split bytes=412
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=283
CPU time spent (ms)=7450
Physical memory (bytes) snapshot=732844032
Virtual memory (bytes) snapshot=8491610112
Total committed heap usage (bytes)=444596224
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=3799556
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Transferred 3.6235 MB in 18.5387 seconds (200.1492 KB/sec)
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Retrieved 3244 records.
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table files
18/05/24 16:00:28 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:28 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 16:00:30 INFO hive.HiveImport:
18/05/24 16:00:30 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 16:00:33 INFO hive.HiveImport: OK
18/05/24 16:00:33 INFO hive.HiveImport: Time taken: 1.786 seconds
18/05/24 16:00:33 INFO hive.HiveImport: Loading data to table test.files
18/05/24 16:00:34 INFO hive.HiveImport: Table test.files stats: [numFiles=4, totalSize=3799556]
18/05/24 16:00:34 INFO hive.HiveImport: OK
18/05/24 16:00:34 INFO hive.HiveImport: Time taken: 0.797 seconds
18/05/24 16:00:34 INFO hive.HiveImport: Hive import complete.
18/05/24 16:00:34 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
18/05/24 16:00:34 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 16:00:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/