1. 程式人生 > >Sqoop匯入MySQL所有表到Hive

Sqoop匯入MySQL所有表到Hive

1、匯入到Hive預設資料庫

[[email protected] sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18
/05/24 15:26:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 18/05/24 15:26:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 18/05/24 15:26:19 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 18/05/24 15:26:19 INFO tool.BaseSqoopTool: delimiters with
--fields-terminated-by, etc. 18/05/24 15:26:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 18/05/24 15:26:20 INFO tool.CodeGenTool: Beginning code generation 18/05/24 15:26:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1 18/05/24 15:26:20 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `files` AS t LIMIT 1 18/05/24 15:26:20 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5 Note: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/files.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 18/05/24 15:26:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/files.jar 18/05/24 15:26:22 WARN manager.MySQLManager: It looks like you are importing from mysql. 18/05/24 15:26:22 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 18/05/24 15:26:22 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 18/05/24 15:26:22 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 18/05/24 15:26:22 INFO mapreduce.ImportJobBase: Beginning import of files 18/05/24 15:26:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 18/05/24 15:26:23 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 18/05/24 15:26:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:26:26 INFO db.DBInputFormat: Using read commited transaction isolation 18/05/24 15:26:26 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `files` 18/05/24 15:26:26 INFO db.IntegerSplitter: Split size: 810; Num splits: 4 from: 1 to: 3244 18/05/24 15:26:26 INFO mapreduce.JobSubmitter: number of splits:4 18/05/24 15:26:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0015 18/05/24 15:26:26 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0015 18/05/24 15:26:26 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0015/ 18/05/24 15:26:26 INFO mapreduce.Job: Running job: job_1526097883376_0015 18/05/24 15:26:33 INFO mapreduce.Job: Job job_1526097883376_0015 running in uber mode : false 18/05/24 15:26:33 INFO mapreduce.Job: map 0% reduce 0% 18/05/24 15:26:39 INFO mapreduce.Job: map 25% reduce 0% 18/05/24 15:26:40 INFO mapreduce.Job: map 100% reduce 0% 18/05/24 15:26:40 INFO mapreduce.Job: Job job_1526097883376_0015 completed successfully 18/05/24 15:26:41 INFO mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=570260 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=412 HDFS: Number of bytes written=3799556 HDFS: Number of read operations=16 HDFS: Number of large read operations=0 HDFS: Number of write operations=8 Job Counters Killed map tasks=1 Launched map tasks=4 Other local map tasks=4 Total time spent by all maps in occupied slots (ms)=16521 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=16521 Total vcore-milliseconds taken by all map tasks=16521 Total megabyte-milliseconds taken by all map tasks=16917504 Map-Reduce Framework Map input records=3244 Map output records=3244 Input split bytes=412 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=274 CPU time spent (ms)=6610 Physical memory (bytes) snapshot=726429696 Virtual memory (bytes) snapshot=8493514752 Total committed heap usage (bytes)=437256192 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=3799556 18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Transferred 3.6235 MB in 17.587 seconds (210.9803 KB/sec) 18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Retrieved 3244 records. 18/05/24 15:26:41 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table files 18/05/24 15:26:41 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1 18/05/24 15:26:41 INFO hive.HiveImport: Loading uploaded data into Hive 18/05/24 15:26:43 INFO hive.HiveImport: 18/05/24 15:26:43 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties 18/05/24 15:26:46 INFO hive.HiveImport: OK 18/05/24 15:26:46 INFO hive.HiveImport: Time taken: 1.753 seconds 18/05/24 15:26:46 INFO hive.HiveImport: Loading data to table default.files 18/05/24 15:26:47 INFO hive.HiveImport: Table default.files stats: [numFiles=4, totalSize=3799556] 18/05/24 15:26:47 INFO hive.HiveImport: OK 18/05/24 15:26:47 INFO hive.HiveImport: Time taken: 0.802 seconds 18/05/24 15:26:47 INFO hive.HiveImport: Hive import complete. 18/05/24 15:26:47 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory. 18/05/24 15:26:47 INFO tool.CodeGenTool: Beginning code generation 18/05/24 15:26:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1 18/05/24 15:26:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5 Note: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/logs.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 18/05/24 15:26:48 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a38c87c70af413cbb1ec75d0b0f5ad34/logs.jar 18/05/24 15:26:48 INFO mapreduce.ImportJobBase: Beginning import of logs 18/05/24 15:26:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:26:49 INFO db.DBInputFormat: Using read commited transaction isolation 18/05/24 15:26:49 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `logs` 18/05/24 15:26:49 INFO db.IntegerSplitter: Split size: 0; Num splits: 4 from: 1 to: 1 18/05/24 15:26:49 INFO mapreduce.JobSubmitter: number of splits:1 18/05/24 15:26:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0016 18/05/24 15:26:50 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0016 18/05/24 15:26:50 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0016/ 18/05/24 15:26:50 INFO mapreduce.Job: Running job: job_1526097883376_0016 18/05/24 15:26:58 INFO mapreduce.Job: Job job_1526097883376_0016 running in uber mode : false 18/05/24 15:26:58 INFO mapreduce.Job: map 0% reduce 0% 18/05/24 15:27:05 INFO mapreduce.Job: map 100% reduce 0% 18/05/24 15:27:05 INFO mapreduce.Job: Job job_1526097883376_0016 completed successfully 18/05/24 15:27:05 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=218298 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=99 HDFS: Number of bytes written=47 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=4480 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=4480 Total vcore-milliseconds taken by all map tasks=4480 Total megabyte-milliseconds taken by all map tasks=4587520 Map-Reduce Framework Map input records=1 Map output records=1 Input split bytes=99 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=65 CPU time spent (ms)=1560 Physical memory (bytes) snapshot=182292480 Virtual memory (bytes) snapshot=2124431360 Total committed heap usage (bytes)=112197632 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=47 18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Transferred 47 bytes in 16.8987 seconds (2.7813 bytes/sec) 18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Retrieved 1 records. 18/05/24 15:27:05 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table logs 18/05/24 15:27:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1 18/05/24 15:27:05 WARN hive.TableDefWriter: Column day had to be cast to a less precise type in Hive 18/05/24 15:27:05 INFO hive.HiveImport: Loading uploaded data into Hive 18/05/24 15:27:07 INFO hive.HiveImport: 18/05/24 15:27:07 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties 18/05/24 15:27:10 INFO hive.HiveImport: OK 18/05/24 15:27:10 INFO hive.HiveImport: Time taken: 1.809 seconds 18/05/24 15:27:10 INFO hive.HiveImport: Loading data to table default.logs 18/05/24 15:27:11 INFO hive.HiveImport: Table default.logs stats: [numFiles=1, totalSize=47] 18/05/24 15:27:11 INFO hive.HiveImport: OK 18/05/24 15:27:11 INFO hive.HiveImport: Time taken: 0.694 seconds 18/05/24 15:27:11 INFO hive.HiveImport: Hive import complete. 18/05/24 15:27:11 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory. [[email protected] sqoop-1.4.7]#
hive> show tables;
OK
files
logs
t1
Time taken: 0.064 seconds, Fetched: 5 row(s)
hive> select count(*) from files;
Query ID = root_20180524152738_798fe3b4-acf1-48a1-a90a-5060fe30269a
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1526097883376_0017, Tracking URL = http://ndoe1:8088/proxy/application_1526097883376_0017/
Kill Command = /opt/hadoop-2.7.5/bin/hadoop job  -kill job_1526097883376_0017
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2018-05-24 15:27:45,426 Stage-1 map = 0%,  reduce = 0%
2018-05-24 15:27:51,784 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.4 sec
2018-05-24 15:27:59,017 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.39 sec
MapReduce Total cumulative CPU time: 8 seconds 390 msec
Ended Job = job_1526097883376_0017
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 3  Reduce: 1   Cumulative CPU: 8.39 sec   HDFS Read: 3816408 HDFS Write: 5 SUCCESS
Total MapReduce CPU Time Spent: 8 seconds 390 msec
OK
3244
Time taken: 21.453 seconds, Fetched: 1 row(s)
hive> 

2、匯入到指定Hive資料庫

預設是匯入到default資料庫中,如果想指定匯入到某個資料庫中,可以使用–hive-database引數

hive> create database test;
OK
Time taken: 0.158 seconds
hive>
[[email protected] sqoop-1.4.7]# bin/sqoop-import-all-tables --connect jdbc:mysql://node1:3306/esdb --username root --password 123456 --hive-import --hive-database test --create-hive-table
Warning: /opt/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
18/05/24 16:00:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/05/24 16:00:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/05/24 16:00:06 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/05/24 16:00:06 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/05/24 16:00:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/05/24 16:00:06 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:06 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/05/24 16:00:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/3036daf1aa7539d081b68965b6d1b061/files.jar
18/05/24 16:00:08 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/05/24 16:00:08 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/05/24 16:00:08 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/05/24 16:00:08 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/05/24 16:00:08 INFO mapreduce.ImportJobBase: Beginning import of files
18/05/24 16:00:08 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/05/24 16:00:09 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/05/24 16:00:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/05/24 16:00:12 INFO db.DBInputFormat: Using read commited transaction isolation
18/05/24 16:00:12 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `files`
18/05/24 16:00:12 INFO db.IntegerSplitter: Split size: 810; Num splits: 4 from: 1 to: 3244
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: number of splits:4
18/05/24 16:00:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526097883376_0019
18/05/24 16:00:12 INFO impl.YarnClientImpl: Submitted application application_1526097883376_0019
18/05/24 16:00:12 INFO mapreduce.Job: The url to track the job: http://bigdata03-test:8088/proxy/application_1526097883376_0019/
18/05/24 16:00:12 INFO mapreduce.Job: Running job: job_1526097883376_0019
18/05/24 16:00:20 INFO mapreduce.Job: Job job_1526097883376_0019 running in uber mode : false
18/05/24 16:00:20 INFO mapreduce.Job:  map 0% reduce 0%
18/05/24 16:00:26 INFO mapreduce.Job:  map 25% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job:  map 100% reduce 0%
18/05/24 16:00:27 INFO mapreduce.Job: Job job_1526097883376_0019 completed successfully
18/05/24 16:00:28 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=570260
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=412
        HDFS: Number of bytes written=3799556
        HDFS: Number of read operations=16
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=8
    Job Counters 
        Killed map tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=17383
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17383
        Total vcore-milliseconds taken by all map tasks=17383
        Total megabyte-milliseconds taken by all map tasks=17800192
    Map-Reduce Framework
        Map input records=3244
        Map output records=3244
        Input split bytes=412
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=283
        CPU time spent (ms)=7450
        Physical memory (bytes) snapshot=732844032
        Virtual memory (bytes) snapshot=8491610112
        Total committed heap usage (bytes)=444596224
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=3799556
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Transferred 3.6235 MB in 18.5387 seconds (200.1492 KB/sec)
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Retrieved 3244 records.
18/05/24 16:00:28 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table files
18/05/24 16:00:28 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `files` AS t LIMIT 1
18/05/24 16:00:28 INFO hive.HiveImport: Loading uploaded data into Hive
18/05/24 16:00:30 INFO hive.HiveImport: 
18/05/24 16:00:30 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/opt/hive-1.2.2/lib/hive-exec-1.2.2.jar!/hive-log4j.properties
18/05/24 16:00:33 INFO hive.HiveImport: OK
18/05/24 16:00:33 INFO hive.HiveImport: Time taken: 1.786 seconds
18/05/24 16:00:33 INFO hive.HiveImport: Loading data to table test.files
18/05/24 16:00:34 INFO hive.HiveImport: Table test.files stats: [numFiles=4, totalSize=3799556]
18/05/24 16:00:34 INFO hive.HiveImport: OK
18/05/24 16:00:34 INFO hive.HiveImport: Time taken: 0.797 seconds
18/05/24 16:00:34 INFO hive.HiveImport: Hive import complete.
18/05/24 16:00:34 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
18/05/24 16:00:34 INFO tool.CodeGenTool: Beginning code generation
18/05/24 16:00:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `logs` AS t LIMIT 1
18/05/24 16:00:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.7.5
Note: /tmp/sqoop-root/compile/