2. 程式人生 > >使用sqoop將mysql 資料匯入hdfs時各種報錯

使用sqoop將mysql 資料匯入hdfs時各種報錯

阿新 • • 發佈：2018-11-11

資訊

  1. 18/06/29 10:45:08 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  2. java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  3. at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:170)

  4. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161)

  5. at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)

  6. at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)

  7. at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:299)

  8. at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)

  9. at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)

  10. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)

  11. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)

  12. at java.security.AccessController.doPrivileged(Native Method)

  13. at javax.security.auth.Subject.doAs(Subject.java:422)

  14. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)

  15. at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)

  16. at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)

  17. at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:200)

  18. at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:173)

  19. at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:270)

  20. at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)

  21. at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)

  22. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)

  23. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)

  24. at org.apache.sqoop.Sqoop.run(Sqoop.java:147)

  25. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

  26. at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)

  27. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)

  28. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)

  29. at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

  30. Caused by: java.lang.RuntimeException: java.sql.SQLException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  31. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:223)

  32. at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168)

  33. ... 26 more

  34. Caused by: java.sql.SQLException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  35. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:127)

  36. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)

  37. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:87)

  38. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:61)

  39. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:71)

  40. at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:85)

  41. at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:440)

  42. at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230)

  43. at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226)

  44. at java.sql.DriverManager.getConnection(DriverManager.java:664)

  45. at java.sql.DriverManager.getConnection(DriverManager.java:247)

  46. at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)

  47. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:216)

  48. ... 27 more

  49. Caused by: com.mysql.cj.exceptions.WrongArgumentException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  50. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  51. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

  52. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

  53. at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

  54. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:59)

  55. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:103)

  56. at com.mysql.cj.conf.DefaultPropertySet.initializeProperties(DefaultPropertySet.java:194)

  57. at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:382)

  58. ... 33 more

  59. Caused by: com.mysql.cj.exceptions.CJException: The connection property 'zeroDateTimeBehavior' acceptable values are: 'CONVERT_TO_NULL', 'EXCEPTION' or 'ROUND'. The value 'convertToNull' is not acceptable.

  60. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  61. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

  62. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

  63. at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

  64. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:59)

  65. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:103)

  66. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:149)

  67. at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:122)

  68. at com.mysql.cj.conf.EnumPropertyDefinition.parseObject(EnumPropertyDefinition.java:64)

  69. at com.mysql.cj.conf.EnumPropertyDefinition.parseObject(EnumPropertyDefinition.java:39)

  70. at com.mysql.cj.conf.AbstractRuntimeProperty.setFromString(AbstractRuntimeProperty.java:96)

  71. at com.mysql.cj.conf.AbstractRuntimeProperty.initializeFrom(AbstractRuntimeProperty.java:91)

  72. at com.mysql.cj.conf.AbstractRuntimeProperty.initializeFrom(AbstractRuntimeProperty.java:75)

  73. at com.mysql.cj.conf.DefaultPropertySet.initializeProperties(DefaultPropertySet.java:191)

  74. ... 34 more

  75. Caused by: java.lang.IllegalArgumentException: No enum constant com.mysql.cj.conf.PropertyDefinitions.ZeroDatetimeBehavior.CONVERTTONULL

  76. at java.lang.Enum.valueOf(Enum.java:238)

  77. at com.mysql.cj.conf.EnumPropertyDefinition.parseObject(EnumPropertyDefinition.java:62)

  78. ... 39 more

原命令是:

sqoop import --connect jdbc:mysql://localhost:3306/FileManager  --username root --password #### --table user

將 fileManager資料倉庫中的表user  匯入到 hdfs

在連線mysql時的連線中加入 

?zeroDateTimeBehavior=EXCEPTION

後來修改為

sqoop import --connect jdbc:mysql://localhost:3306/FileManager?zeroDateTimeBehavior=EXCEPTION --username root --password #### --table user

 

就可以執行了

 

問題1：sqoop import匯入時報java.lang.ClassNotFoundException: org.json.JSONObject 錯誤

 

 

  1. [[email protected] lib]# sqoop import --connect jdbc:mysql://10.1.32.8:3306/test --username sqoop --password sqoop --table t1 -m 1

  2. 16/06/07 08:48:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0

  3. 16/06/07 08:48:59 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

  4. 16/06/07 08:48:59 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

  5. 16/06/07 08:48:59 INFO tool.CodeGenTool: Beginning code generation

  6. 16/06/07 08:48:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t1` AS t LIMIT 1

  7. 16/06/07 08:48:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t1` AS t LIMIT 1

  8. 16/06/07 08:48:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce

  9. Note: /tmp/sqoop-root/compile/07751371c513f90a6377d7b482c4a910/t1.java uses or overrides a deprecated API.

  10. Note: Recompile with -Xlint:deprecation for details.

  11. 16/06/07 08:49:01 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/07751371c513f90a6377d7b482c4a910/t1.jar

  12. 16/06/07 08:49:01 WARN manager.MySQLManager: It looks like you are importing from mysql.

  13. 16/06/07 08:49:01 WARN manager.MySQLManager: This transfer can be faster! Use the --direct

  14. 16/06/07 08:49:01 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.

  15. 16/06/07 08:49:01 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)

  16. 16/06/07 08:49:01 INFO mapreduce.ImportJobBase: Beginning import of t1

  17. Exception in thread "main" <span style="color:#ff0000;">java.lang.NoClassDefFoundError: org/json/JSONObject</span>

  18. at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:42)

  19. at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:742)

  20. at org.apache.sqoop.mapreduce.JobBase.putSqoopOptionsToConfiguration(JobBase.java:369)

  21. at org.apache.sqoop.mapreduce.JobBase.createJob(JobBase.java:355)

  22. at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:249)

  23. at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)

  24. at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)

  25. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)

  26. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)

  27. at org.apache.sqoop.Sqoop.run(Sqoop.java:143)

  28. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

  29. at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

  30. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)

  31. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)

  32. at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

  33. Caused by: <span style="color:#ff0000;">java.lang.ClassNotFoundException: org.json.JSONObject</span>

  34. at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

  35. at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

  36. at java.security.AccessController.doPrivileged(Native Method)

  37. at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

  38. at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

  39. at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

  40. at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

  41. ... 15 more

 

 

解決：

這是因為sqoop缺少java-json.jar包.

如下面連結所說：

http://stackoverflow.com/questions/27504508/java-lang-noclassdeffounderror-org-json-jsonobject

 

The Exception it self says itall java.lang.ClassNotFoundException: org.json.JSONObject

You have not added thenecessary jar file which will be having org.json.JSONObject class to yourclasspath.

 

下載java-json.jar包：

http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm

 

把java-json.jar新增到../sqoop/lib目錄：

# cp java-json.jar/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/sqoop/lib

[[email protected] lib]# pwd

/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/sqoop/lib

[[email protected] lib]#

[[email protected] lib]$ ll java-json.jar

-rw-r--r-- 1 root root 84697 Oct 16  2013 java-json.jar

 


問題2：root使用者寫入HDFS檔案錯誤 "Permission denied: user=root"

 

 

  1. [[email protected] lib]# sqoop import --connect jdbc:mysql://10.1.32.8:3306/test --username sqoop --password sqoop --table t1 -m 1

  2. 16/06/07 08:49:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0

  3. 16/06/07 08:49:50 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

  4. ... ...

  5. 16/06/07 08:49:52 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)

  6. 16/06/07 08:49:52 INFO mapreduce.ImportJobBase: Beginning import of t1

  7. 16/06/07 08:49:53 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar

  8. 16/06/07 08:49:53 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

  9. 16/06/07 08:49:53 INFO client.RMProxy: Connecting to ResourceManager at hadoop0.hadoop.com/10.1.32.239:8032

  10. 16/06/07 08:49:54 <span style="color:#ff0000;">WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x</span>

  11. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)

  12. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)

  13. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:242)

  14. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:169)

  15. at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)

  16. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6590)

  17. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6572)

  18. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6524)

  19. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4322)

  20. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4292)

  21. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4265)

  22. at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:867)

  23. at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:322)

  24. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:603)

  25. at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

  26. at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

  27. at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)

  28. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)

  29. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)

  30. at java.security.AccessController.doPrivileged(Native Method)

  31. at javax.security.auth.Subject.doAs(Subject.java:415)

  32. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

  33. at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

  34.  

  35. 16/06/07 08:49:54 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

  36. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)

  37. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)

  38. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:242)

  39. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:169)

  40. at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)

  41. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6590)

  42. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6572)

  43. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6524)

  44. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4322)

  45. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4292)

  46. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4265)

  47. at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:867)

  48. at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:322)

  49. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:603)

  50. at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

  51. at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

  52. at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)

  53. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)

  54. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)

  55. at java.security.AccessController.doPrivileged(Native Method)

  56. at javax.security.auth.Subject.doAs(Subject.java:415)

  57. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

  58. at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

  59.  

  60. Caused by: <span style="color:#ff0000;">org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x</span>

  61. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)

  62. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)

  63. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:242)

  64. at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:169)

  65. at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)

  66. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6590)

  67. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6572)

  68. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6524)

  69. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4322)

  70. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4292)

  71. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4265)

  72. at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:867)

  73. at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:322)

  74. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:603)

  75. at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

  76. at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

  77. at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)

  78. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)

  79. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)

  80. at java.security.AccessController.doPrivileged(Native Method)

  81. at javax.security.auth.Subject.doAs(Subject.java:415)

  82. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

  83. at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

  84.  

  85. at org.apache.hadoop.ipc.Client.call(Client.java:1471)

  86. at org.apache.hadoop.ipc.Client.call(Client.java:1408)

  87. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)

  88. at com.sun.proxy.$Proxy15.mkdirs(Unknown Source)

  89. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:544)

  90. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  91. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  92. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  93. at java.lang.reflect.Method.invoke(Method.java:606)

  94. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)

  95. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)

  96. at com.sun.proxy.$Proxy16.mkdirs(Unknown Source)

  97. at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3082)

  98. ... 28 more


 

解決：
因為在使用sqoop匯入HDFS檔案時，使用的是root使用者，沒有寫hdfs檔案的許可權。
CDH安裝時建立了hdfs使用者，應使用hdfs使用者登入，再時行sqoop匯入
[[email protected] ~]# su - hdfs
[[email protected] ~]$ pwd
/var/lib/hadoop-hdfs
[[email protected] ~]$ ls
t1.java

 

問題3：其它hadoop節點不連線MySql "is not allowed to connect to this MySQL server"

 

 

  1. [[email protected] ~]$ sqoop import --connect jdbc:mysql://10.1.32.8:3306/test --username sqoop --password sqoop --table t1 -m 1

  2. 16/06/07 08:51:52 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0

  3. ......

  4. 16/06/07 08:52:00 INFO mapreduce.Job: The url to track the job: http://hadoop0.hadoop.com:8088/proxy/application_1464249387420_0001/

  5. 16/06/07 08:52:00 INFO mapreduce.Job: Running job: job_1464249387420_0001

  6. 16/06/07 08:52:08 INFO mapreduce.Job: Job job_1464249387420_0001 running in uber mode : false

  7. 16/06/07 08:52:08 INFO mapreduce.Job: map 0% reduce 0%

  8. 16/06/07 08:52:14 INFO mapreduce.Job: Task Id : attempt_1464249387420_0001_m_000000_0, Status : FAILED

  9. <span style="color:#ff0000;">Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: null, message from server: "Host 'hadoop4.hadoop.com' is not allowed to connect to this MySQL server"</span>

  10. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

  11. at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)

  12. at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

  13. at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)

  14. at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

  15. at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

  16. at java.security.AccessController.doPrivileged(Native Method)

  17. at javax.security.auth.Subject.doAs(Subject.java:415)

  18. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

  19. at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  20. Caused by: java.lang.RuntimeException: java.sql.SQLException: null, message from server: "Host 'hadoop4.hadoop.com' is not allowed to connect to this MySQL server"

  21. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)

  22. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)

  23. ... 9 more

  24. Caused by: java.sql.SQLException: null, message from server: "Host 'hadoop4.hadoop.com' is not allowed to connect to this MySQL server"

 

解決：
這是因為客戶端沒有訪問mysql的許可權，修改sqoop使用者的客戶端訪問許可權。
--登入mysql伺服器
[[email protected] ~]# mysql -uroot -proot 
mysql> grant all privileges on *.* to 'sqoop'@'%' identified by 'sqoop' with grant option;
Query OK, 0 rows affected (0.00 sec)

相關推薦

使用sqoop將mysql 資料匯入hdfs時各種報錯

資訊 18/06/29 10:45:08 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLExcepti

通過sqoop將mysql資料匯入到hive中進行計算示例

hive計算示例 先將資料通過sqoop從mysql匯入hive，在hive執行mysql的查詢語句，得到與mysql一樣的執行結果 步驟： mysql資料準備 account賬號表 detail收支資料表 CREATE TABLE

sqoop將Mysql資料匯入Hive中

注：筆者用的是sqoop1.4.6版本 一、將Mysql資料匯入Hive中 命令： sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --con

使用sqoop將oracle資料匯入hdfs叢集

叢集環境：     hadoop1.0.0     hbase0.92.1     zookeeper3.4.3     hive0.8.1     sqoop-1.4.1-incubating__hadoop-1.0.0.tar 首先，當然前提是hadoop叢集環境已經

在hadoop 裡安裝 sqoop 並把mysql資料匯入hdfs

hadoop 2.6.0 sqoop：sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz 然後把sqoop 解壓。這裡解壓發生的問題是， tar: /home/luis：歸檔中找不到 tar: 由於前次錯誤，將以上次的錯誤狀態退出 ta

利用sqoop將hive資料匯入匯出資料到mysql

執行環境  centos 5.6   hadoop  hive sqoop是讓hadoop技術支援的clouder公司開發的一個在關係資料庫和hdfs,hive之間資料匯入匯出的一個工具 在使用過程中可能遇到的問題： sqoop依賴zookeeper，所以必須配置ZOOK

使用sqoop將hive資料匯入mysql例項

1.環境配置 CentOS6.5 hadoop2.2 jdk1.7.0 sqoop1.4.4 zookeeper3.4.5 Mysql 14.14 2.在mysql上建立表 先按照需求在

利用sqoop將hive資料匯入Oracle中（踩的坑）

教程很多，這裡只說踩過的坑 1.下載sqoop時，還得下一個bin的包，拿到sqoop-1.4.6.jar 的包，放到hadoop的lib目錄下 2.匯入oracle，執行程式碼時，使用者名稱和表名必須大寫！且資料庫建表時表名必須大寫！  示例程式碼： sqoop expo

mongodb將mysql資料匯入

1、首先將資料從mysql資料庫匯出為xls檔案 SELECT * FROM user INTO OUTFILE "F:\1.xls" 2、notepad++開啟，用utf8編碼儲存資料   3、使用wps或者cvs開啟檔案，並以.cvs格式儲存資料   4、在w

利用sqoop將hive資料匯入Oracle中

首先： 如oracle則執行sqoop list-databases --connect jdbc:oracle:thin:@//192.168.27.235:1521/ORCL --username DATACENTER -P 來測試是否能正確連線資料庫 　如mysql則執行sq

用sqoop將oracle資料匯入Hbase 使用筆記

網上已經有很多關於這方面的資料，但是我在使用過程中也遇見了不少問題 1. sqoop 的環境我沒有自己搭建  直接用的公司的 2. oracle 小白怕把公司環境弄壞了，自己用容器搭建了一個 docker pull docker.io/wnameless/oracle-xe

sqoop因為連線資料寫了localhost而報錯

1.問題 最開始執行sqoop： bin/sqoop import \ --connect jdbc:mysql://localhost:3306/sqoop \ --username root \ --password root \ --table emp --m 1 報錯

python使用json時各種報錯問題

1、str轉換成json時使用json.dumps出現亂碼（原始字元編碼） >>> import json >>> srt = js = json.loads('{"haha": "哈哈"}') >>> print json.dumps(js

利用Sqoop將MySQL海量測試資料匯入HDFS和HBase

宣告：作者原創，轉載註明出處。 一、安裝Sqoop 1、下載sqoop,解壓、資料夾重新命名 wget http://mirror.bit.edu.cn/apache/sqoop/1.4.6/sqoop-1.4.6.bin_

使用sqoop將mysql中的資料匯入Hive時報錯

1.使用sqoop在hive中建立一個和mysql中資料結構一樣的表時報錯  ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set [em

用sqoop將mysql的資料匯入到hive表中，原理分析

Sqoop 將 Mysql 的資料匯入到 Hive 中 準備Mysql 資料 如圖所示，準備一張表，資料隨便造一些，當然我這裡的資料很簡單。 編寫命令 編寫引數檔案 個人習慣問題，我喜歡把引數寫到檔案裡，然後再命令列引用。 vim mysql-info， #

通過sqoop將MySQL資料庫中的資料匯入Hbase

從接觸到大資料到成功的實現一個功能期間走了不少彎路也踩了不少坑，這裡作為我的學習筆記也可以作為小白們的前車之鑑，少走彎路，有不正確之處，望指出 環境準備： hadoop、hbase、sqoop、mys

用sqoop將mysql的資料匯入到hive表中

用sqoop將mysql的資料匯入到hive表中 1：先將mysql一張表的資料用sqoop匯入到hdfs中 準備一張表    需求 將 bbs_product 表中的前100條資料導 匯出來  只要id  brand_id和 na

將excel資料匯入到mysql資料庫

框架用的是thinkphp3.2框架 //將Excel匯入到資料庫的思想是：先將Excel中的資料轉為php陣列，然後執行addAll操作 public function upload() { ini_set('memory_limit', '1024M'); if (!empty(

資料匯入終章：如何將HBase的資料匯入HDFS？

我們的最終目標是將資料匯入Hadoop，在之前的章節中，我們介紹瞭如何將傳統關係資料庫的資料匯入Hadoop，本節涉及到了HBase。HBase是一種實時分散式資料儲存系統，通常位於與Hadoop叢集相同的硬體上，或者與Hadoop叢集緊密相連，能夠直接在MapReduce中使用HBase資料，或將