我有一个mysql表“customers”,并尝试使用sqoop import将数据从mysql导入hdfs位置。下面是我在我的机器上安装的版本:sqoop版本:1.4.6 hive版本:2.3.0 hadoop版本:2.8.1
sqoop导入命令:
sqoop import --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234 --table customers -m 1 --target-dir /user/hduser/sqoop_import/customers1/
并显示以下类别NotFountException:
Tue Oct 31 09:57:21 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.17/10/31 09:57:21 INFO db.DBInputFormat: Using read commited transaction isolation 17/10/31 09:57:21 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
17/10/31 09:57:21 INFO mapred.LocalJobRunner: map task executor complete.
17/10/31 09:57:22 WARN mapred.LocalJobRunner: job_local1437452057_0001java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:237)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:263) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:515)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class customers not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
... 12 more
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 running in uber mode : false 17/10/31 09:57:22 INFO mapreduce.Job: map 0% reduce 0%
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 failed with state FAILED due to: NA 17/10/31 09:57:22 INFO mapreduce.Job: Counters: 0
17/10/31 09:57:22 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 4.105 seconds (0 bytes/sec)
17/10/31 09:57:22 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Retrieved 0 records.
17/10/31 09:57:22 ERROR tool.ImportTool: Error during import: Import job failed!
但是,当我尝试使用sqoop命令列出表时,它运行良好,可以显示customers表。sqoop命令:
sqoop list-tables --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234;
输出正确显示,如图所示:
17/10/31 10:07:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Tue Oct 31 10:07:09 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. customers
可能的问题是,为什么sqoop import from table无法识别mysql中的表。请在同样的问题上帮助我。
提前谢谢。
1条答案
按热度按时间f8rj6qna1#
使用
--bindir
选项并指向当前工作目录 --宾迪尔 选项给出 $sqoop\u主页/库抱歉,回复太晚。请根据最新评论更新答案。请检查以下内容:
当sqoop进程在cmd中运行时,需要检查jar文件:
Writing jar file:/tmp/sqoop.../*.jar
使用导入命令:sqoop import -fs local -jt local -libjars /tmp/..../*.jar --connect "jdbc:mysql://example.com/sqoop" --username <>--password <>--table customers
验证为sqoop导入生成的java文件。它应该有与正常导入的表同名。
sqoop codegen --connect <""> --username <> --password <> --table customers
希望这有帮助!