2. 程式人生 > 其它 >Sqoop從MySQL向Hive增量式匯入資料報錯：Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

Sqoop從MySQL向Hive增量式匯入資料報錯：Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

阿新 • • 發佈：2021-11-06

1、問題描述：

（1）問題示例：

Step1:建立作業：

[Hadoop@master TestDir]$ sqoop job \
> --create myjob_1 \
> -- import \
> --connect "jdbc:mysql://master:3306/source?useSSL=false&user=Hive&password=******" \
> --table sales_order \
> --columns "order_number,customer_number,product_code,order_date,entry_date,order_amount" \


> --where "entry_date < current_date()" \
> --hive-import \
> --hive-table rds.sales_order \
> --incremental append \
> --check-column entry_date \
> --last-value '1900-01-01'
2021-11-06 15:08:48,003 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2021-11-06 15:08:48,242 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
2021-11-06 15:08:48,242 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 12 more

Step2:檢視作業：
[Hadoop@master TestDir]$ sqoop job --list
2021-11-06 15:09:01,663 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Available jobs:
myjob_1

Step3:檢視時間戳：
[Hadoop@master TestDir]$ sqoop job --show myjob_1 |grep last.value
2021-11-06 15:09:10,169 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
at org.apache.sqoop.tool.JobTool.showJob(JobTool.java:261)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:302)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 10 more

Step4:提交作業：
[Hadoop@master TestDir]$ sqoop job --exec myjob_1
2021-11-06 15:09:20,062 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
--table or --query is required for import. (Or use sqoop import-all-tables.)
Try --help for usage instructions.

Step5:二次檢視時間戳：
[Hadoop@master TestDir]$ sqoop job --show myjob_1 |grep last.value
2021-11-06 15:09:28,496 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
at org.apache.sqoop.tool.JobTool.showJob(JobTool.java:261)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:302)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 10 more

（2）問題綜述：

通過以上步驟示例，Sqoop從MySQL向Hive進行增量式匯入資料報錯，問題的核心在於：

Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

解決該問題，是關鍵。

2、問題剖析：

（1）參考：https://www.cnblogs.com/QuestionsZhang/p/10082735.html

（2）剖析：根據參考，以及本示例所報出的問題，基本可以肯定是因為Sqoop安裝路徑下lib目錄缺少java-json.jar。因此只需要尋找一個Java-json.jar放入Sqoop安裝路徑中的lib目錄下即可。筆者從最終測試結果可知，此為正解。

3、解決方案：

（1）下載java-json.jar：

下載地址：http://www.java2s.com/Code/JarDownload/java-json/java-json.jar.zip

（2）二次測試：

Step1:建立作業：

[Hadoop@master tmp]$ sqoop job \
> --create myjob_1 \
> -- import \
> --connect "jdbc:mysql://master:3306/source?useSSL=false&user=Hive&password=******" \
> --table sales_order \
> --columns "order_number,customer_number,product_code,order_date,entry_date,order_amount" \
> --where "entry_date < current_date()" \
> --hive-import \
> --hive-table rds.sales_order \
> --incremental append \
> --check-column entry_date \
> --last-value '1900-01-01'
2021-11-06 15:49:52,350 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2021-11-06 15:49:52,632 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
2021-11-06 15:49:52,632 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.

Step2:檢視作業：

[Hadoop@master tmp]$ sqoop job --list
2021-11-06 15:50:10,531 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Available jobs:
myjob_1

Step3:檢視時間戳：

[Hadoop@master tmp]$ sqoop job --show myjob_1 |grep last.value
2021-11-06 15:50:29,832 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
incremental.last.value = 1900-01-01

Step4:提交作業：

[Hadoop@master tmp]$ sqoop job --exec myjob_1
2021-11-06 16:05:23,188 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2021-11-06 16:05:23,815 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2021-11-06 16:05:23,851 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2021-11-06 16:05:27,962 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sales_order` AS t LIMIT 1
2021-11-06 16:05:29,150 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sales_order` AS t LIMIT 1
......(此處有省略)
2021-11-06 16:09:35,112 INFO hive.HiveImport: OK
2021-11-06 16:09:35,273 INFO hive.HiveImport: Time taken: 1.611 seconds
2021-11-06 16:09:35,274 INFO hive.HiveImport: 2021-11-06 16:09:35,273 INFO [af634dbf-7808-4849-bd2e-f2b12b7435a8 main] CliDriver (SessionState.java:printInfo(1227)) - Time taken: 1.611 seconds
2021-11-06 16:09:35,274 INFO hive.HiveImport: 2021-11-06 16:09:35,273 INFO [af634dbf-7808-4849-bd2e-f2b12b7435a8 main] conf.HiveConf (HiveConf.java:getLogIdVar(5040)) - Using the default value passed in for log id: af634dbf-7808-4849-bd2e-f2b12b7435a8
2021-11-06 16:09:35,274 INFO hive.HiveImport: 2021-11-06 16:09:35,273 INFO [af634dbf-7808-4849-bd2e-f2b12b7435a8 main] session.SessionState (SessionState.java:resetThreadName(452)) - Resetting thread name to main
2021-11-06 16:09:35,274 INFO hive.HiveImport: 2021-11-06 16:09:35,274 INFO [main] conf.HiveConf (HiveConf.java:getLogIdVar(5040)) - Using the default value passed in for log id: af634dbf-7808-4849-bd2e-f2b12b7435a8
2021-11-06 16:09:35,319 INFO hive.HiveImport: 2021-11-06 16:09:35,318 INFO [main] session.SessionState (SessionState.java:dropPathAndUnregisterDeleteOnExit(885)) - Deleted directory: /user/hive/tmp/grid/af634dbf-7808-4849-bd2e-f2b12b7435a8 on fs with scheme hdfs
2021-11-06 16:09:35,319 INFO hive.HiveImport: 2021-11-06 16:09:35,319 INFO [main] session.SessionState (SessionState.java:dropPathAndUnregisterDeleteOnExit(885)) - Deleted directory: /tmp/hive/Local/af634dbf-7808-4849-bd2e-f2b12b7435a8 on fs with scheme file
2021-11-06 16:09:35,319 INFO hive.HiveImport: 2021-11-06 16:09:35,319 INFO [main] metastore.HiveMetaStoreClient (HiveMetaStoreClient.java:close(600)) - Closed a connection to metastore, current connections: 1
2021-11-06 16:09:35,389 INFO hive.HiveImport: Hive import complete.
2021-11-06 16:09:35,392 INFO hive.HiveImport: Export directory is empty, removing it.
2021-11-06 16:09:35,408 INFO tool.ImportTool: Saving incremental import state to the metastore
2021-11-06 16:09:35,454 INFO tool.ImportTool: Updated data for job: myjob_1

Step5:二次檢視時間戳：

[Hadoop@master tmp]$ sqoop job --show myjob_1 |grep last.value
2021-11-06 16:09:45,037 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
incremental.last.value = 2016-06-30 17:26:24.0

二次檢視時間戳的目的是驗證：Sqoop是不是成功將MySQL中資料採用增量式匯入Hive中，此外還可以在Hive檢視資料進行驗證。

相關推薦

Sqoop從MySQL向Hive增量式匯入資料報錯：Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/json/JSONObject

1、問題描述： （1）問題示例： Step1:建立作業： [Hadoop@master TestDir]$ sqoop job \\> --create myjob_1 \\> -- import \\> --connect \"jdbc:mysql://master:3306/source?useSSL=false&user=Hiv

presto-cli 查詢hive資料報錯：Catalog ‘hive‘ does not exist

錯誤描述 使用prsto-cli進行hive查詢時,報錯Catalog ‘hive‘ does not exist 同時視覺化介面無法查詢到hive的catalog

MyBatis向Oracle資料庫插入NULL資料報錯

技術標籤：java經驗分享javamybatis 原文連結 https://blog.wanvale.com/archives/86/ 摘要 報錯內容如下：

Sqoop從關係型資料庫往HDFS上匯入及匯出檔案

將關係型資料庫(RDBMS)匯入到HDFS         關係型資料庫（RDBMS）匯入到Hive   sqoop引數筆記沒有   （pdf格式的）

sqoop從hdfs上匯出資料到MySQL報錯：ClassNotFoundException:allclo

技術標籤：大資料基礎——Hadoopsqoophadoophdfs大資料 Sqoop操作前提：關閉防火牆+開啟叢集保證其正常。

解決sqoop從postgresql拉資料,報錯TCP/IP連線的問題

問題： sqoop從postgresql拉資料，在執行到mapreduce時報錯Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections

MySQL匯入檔案報錯[HY000][1290] The MySQL server is running with the --secure-file-priv option so it cannot

技術標籤：筆記工具MySQL 問題現象： 完整的報錯資訊：[HY000][1290] The MySQL server is running with the --secure-file-priv option so it cannot execute this statement

Sqoop報錯：Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xF0\xA4\x8F\x81' 再見亂碼：5分鐘讀懂MySQL字符集設定 

sqoop從hive匯出到mysql報錯如下： Caused by: java.sql.SQLException: Incorrect string value: \'\\xF0\\xA4\\x8F\\x81\' for column \'role_name\' at row 68

Django重灌mysql後啟動報錯：No module named ‘MySQLdb’的解決方法

發現問題 最近由於解除安裝Mysql時將很多相關依賴包都解除安裝了，重灌mysql後啟動django出現如下錯誤：

Linux連線mysql報錯：Access denied for user ‘root’@‘localhost’(using password: YES)的解決方法

前言 最近在linux連線mysql /usr/local/mysql/bin/mysql -uroot -p 輸入密碼出現Access denied for user \'root\'@\'localhost\'(using password: YES)錯誤。下面話不多說了，來一起看看詳細的解決方法吧

匯入tensorflow時報錯：cannot import name 'abs'的解決

一，問題背景 作者在用tensorflow做實驗時，import tensorflow忽然報錯：cannot import name \'abs\'。錯誤情況如下所示：

解決IDEA連線mysql報錯：Server returns invalid timezone. Go to &#039;Advanced&#039; tab and set &#039;server

原因： MySQL驅動jar中的預設時區是UTC。 UTC代表的是全球標準時間 ，但是我們使用的時間是北京時區也就是東八區，領先UTC八個小時。

android studio 3.6.1匯入專案報錯提示無法下載classpath裡的內容

報錯： Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException: Could not resolve all artifacts for configuration ‘:classpath\'.

eclipse匯入工程報錯問題專案或者檔案有紅叉的解決方案

1.Java的JDK或者Tomcat版本不一致 像這樣的我們首先找到專案---->Build Path--->Config BuildPath，將找不到的JDK或者Tomcat進行刪除和新增你電腦上已有的版本。新增好記得儲存一下。

pyspark調mysql報錯：“java.lang.ClassNotFoundException: com.mysql.jdbc.Driver”

問題描述 在做http://dblab.xmu.edu.cn/blog/2335/實驗時，做到最後一部分程式碼的第20行時如下圖。

MySQL啟動報錯：發生系統錯誤 5。拒絕訪問。

1、錯誤樣式 錯誤樣式，如下圖所示： 2、導致原因 未使用管理員角色進行此操作；

Eclipse匯入專案報錯問題解決方案

使用eclipse匯入一些專案是會出現專案前有叉號的問題 總結一下解決方法： 1.修改自己的Build Path，右鍵專案名：

Mac 啟動mysql報錯：Can't connect to local MySQL server through socket '/tmp/mysql.sock'

背景： 許久不開啟本地mysql連線，今天突然連線不上了，報錯： ➜~ mysql -version

service mysqld start 啟動mysql服務報錯：/etc/init.d/mysqld: line 263: @HOSTNAME@: command not found ; /etc/init.d/mysqld: line 276: cd: @prefix@: No such file or directory

問題： #編譯安裝完成後複製啟動指令碼“cp -a mysql.server.sh /etc/init.d/mysqld”後，啟動服務報錯

HIVE報錯：Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

執行insert into table video_orc select * from video_ori;時報錯 檢視hive日誌發現具體報錯資訊如下：