sqoop在工作流中失败

wecizke3  于 2021-06-03  发布在  Sqoop
关注(0)|答案(0)|浏览(355)

当下面的sqoop导入在命令shell中运行时,效果很好。

import --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username retail_dba --password cloudera -m 1 --table categories --hive-database retail_stage --hive-table categories --fields-terminated-by "|" --hive-import

但在工作流中运行的同一语句失败,并出现以下错误

>>> Invoking Sqoop command line now >>>

2019-02-04 11:46:18,411 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,609 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.6-cdh5.13.0
2019-02-04 11:46:18,664 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
2019-02-04 11:46:18,696 [main] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,936 [main] INFO  org.apache.sqoop.manager.MySQLManager  - Preparing to use a MySQL streaming resultset.
2019-02-04 11:46:18,951 [main] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
2019-02-04 11:46:20,510 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,555 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,565 [main] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
2019-02-04 11:46:25,907 [main] ERROR org.apache.sqoop.tool.ImportTool  - Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:222)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:187)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:170)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

<<< Invocation of Sqoop command completed <<<

No child hadoop job is executed.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

记住,当 list-databases 命令运行与色调工作流,工作良好。

cloudera快速启动vm(docker映像)详细信息

版本: Cloudera Express 5.13.0 (#55 built by jenkins on 20171002-1719 git: bd657e597e6743c458ee2c9aabe808b7c972981c) java虚拟机名称: Java HotSpot(TM) 64-Bit Server VM java虚拟机供应商: Oracle Corporation java版本: 1.7.0_67 实际上,在oozie中作为sqoop操作放置的任何命令都会失败。
下面是如何在docker映像中启动cloudera quickstart vm
启动cloudera快速启动容器: docker run --hostname=quickstart.cloudera --privileged=true -t -i -v /Users/Yunus/Documents/ClouderaShare:/src --publish-all=true -p 8888:8888 -p 8020:8020 -p 8032:8032 -p 7180:7180 -p 80:80 -p 50070:50070 -p 11000:11000 -p 21050:21050 -p 8088:8088 -p 8042:8042 cloudera-5-13 /usr/bin/docker-quickstart 启动cloudera manager: home/cloudera/cloudera-manager --express 修复时钟偏移问题: /etc/init.d/ntpd start usr/share/java/目录中已存在mysql连接器: sudo -u hdfs hadoop fs -put usr/share/java/mysql-connector-java-5.1.34-bin.jar /user/oozie/share/lib/lib_20171023234839/sqoop

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题