sqoop导入到配置单元在某个点无限期挂起

eimct9ow  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(315)

我尝试使用sqoop import将mysql表导入到hive，但是在命令执行之后，cli保持平静，没有任何事情发生，并且无限期地挂起。以下是命令和问题的详细信息。。

[cloudera@quickstart bin]$ sqoop create-hive-table --connect jdbc:mysql://10.X.X.XX:XXXX/rkdb --username root -P --table employee --hive-table emps

警告：/usr/lib/sqoop/./acumulo不存在！accumulo进口将失败。请将$accumulo\u home设置为accumulo安装的根目录。17/08/30 22:20:02 info sqoop.sqoop:运行sqoop版本：1.4.6-cdh5.12.0输入密码：17/08/30 22:20:05 info tool.basesqooptool:使用配置单元特定的分隔符进行输出。您可以覆盖17/08/30 22:20:05 info tool.basesqooptool:分隔符--字段以等结尾。17/08/30 22:20:05 info manager.mysqlmanager:准备使用mysql流式处理结果集。17/08/30 22:20:06 info manager.sqlmanager:执行sql语句：从中选择t.* employee as t limit 1 17/08/30 22:20:06 info manager.sqlmanager:执行sql语句：从中选择t.* employee as t limit 1 17/08/30 22:20:08 info hive.hiveimport:正在将上载的数据加载到hive
使用中的配置初始化日志记录jar:file：/usr/lib/hive/lib/hive-common-1.1.0-cdh5.12.0.jar/hive-log4j.属性
任何线索都非常感谢。
edit：在命令中包含--verbose以获取详细记录：

Note: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/08/31 01:34:13 DEBUG orm.CompilationManager: Could not rename /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.java to /usr/lib/spark/examples/lib/./employee.java
17/08/31 01:34:13 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.jar
17/08/31 01:34:13 DEBUG orm.CompilationManager: Scanning for .class files in directory: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee$1.class -> employee$1.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee$2.class -> employee$2.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee$3.class -> employee$3.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee$4.class -> employee$4.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee$FieldSetterCommand.class -> employee$FieldSetterCommand.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.class -> employee.class
17/08/31 01:34:13 DEBUG orm.CompilationManager: Finished writing jar file /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.jar
17/08/31 01:34:13 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/08/31 01:34:13 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/08/31 01:34:13 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/08/31 01:34:13 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/08/31 01:34:13 DEBUG manager.MySQLManager: Rewriting connect string to jdbc:mysql://10.0.2.15:3306/rkdb?zeroDateTimeBehavior=convertToNull
17/08/31 01:34:13 INFO mapreduce.ImportJobBase: Beginning import of employee
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Checking for existing class: employee
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Attempting to load jar through URL: jar:file:/tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.jar!/
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Previous classloader is sun.misc.Launcher$AppClassLoader@7d487b8b
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Testing class in jar: employee
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Loaded jar into current JVM: jar:file:/tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.jar!/
17/08/31 01:34:13 DEBUG util.ClassLoaderStack: Added classloader for jar /tmp/sqoop-cloudera/compile/5b58dd4e681df3737c3f8ce4f32013ac/employee.jar: java.net.FactoryURLClassLoader@2ea3741
17/08/31 01:34:16 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/08/31 01:34:16 DEBUG db.DBConfiguration: Securing password into job credentials store
17/08/31 01:34:16 DEBUG mapreduce.DataDrivenImportJob: Using table class: employee
17/08/31 01:34:16 DEBUG mapreduce.DataDrivenImportJob: Using InputFormat: class com.cloudera.sqoop.mapreduce.db.DataDrivenDBInputFormat
17/08/31 01:34:19 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.6-cdh5.12.0.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/share/java/mysql-connector-java-5.1.34-bin.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.6-cdh5.12.0.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.6-cdh5.12.0.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-lang3-3.4.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-mapper-asl-1.8.8.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-hadoop.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/kite-hadoop-compatibility.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-logging-1.1.3.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/slf4j-api-1.7.5.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/opencsv-2.3.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/xz-1.0.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-jexl-2.1.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-format.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-compress-1.4.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-avro.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-codec-1.4.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-encoding.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-jackson.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/kite-data-core.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-core-2.3.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/kite-data-mapreduce.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-column.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro-mapred-hadoop2.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-databind-2.3.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/paranamer-2.3.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/snappy-java-1.0.4.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/hsqldb-1.8.0.10.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/kite-data-hive.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-io-1.4.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-annotations-2.3.1.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-core-asl-1.8.8.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/parquet-common.jar
17/08/31 01:34:19 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/fastutil-6.3.jar
17/08/31 01:34:20 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/10.0.2.15:8032
17/08/31 01:34:23 WARN hdfs.DFSClient: Caught exception 
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1281)
    at java.lang.Thread.join(Thread.java:1355)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:952)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:690)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:879)
17/08/31 01:34:25 WARN hdfs.DFSClient: Caught exception 
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1281)
    at java.lang.Thread.join(Thread.java:1355)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:952)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:690)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:879)
17/08/31 01:34:26 DEBUG db.DBConfiguration: Fetching password from job credentials store
17/08/31 01:34:26 INFO db.DBInputFormat: Using read commited transaction isolation
17/08/31 01:34:26 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `employee`
17/08/31 01:34:26 INFO db.IntegerSplitter: Split size: 125; Num splits: 4 from: 100 to: 600
17/08/31 01:34:26 DEBUG db.IntegerSplitter: Splits: [                         100 to                          600] into 4 parts
17/08/31 01:34:26 DEBUG db.IntegerSplitter:                          100
17/08/31 01:34:26 DEBUG db.IntegerSplitter:                          225
17/08/31 01:34:26 DEBUG db.IntegerSplitter:                          350
17/08/31 01:34:26 DEBUG db.IntegerSplitter:                          475
17/08/31 01:34:26 DEBUG db.IntegerSplitter:                          600
17/08/31 01:34:26 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 100' and upper bound '`id` < 225'
17/08/31 01:34:26 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 225' and upper bound '`id` < 350'
17/08/31 01:34:26 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 350' and upper bound '`id` < 475'
17/08/31 01:34:26 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 475' and upper bound '`id` <= 600'
17/08/31 01:34:26 WARN hdfs.DFSClient: Caught exception 
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1281)
    at java.lang.Thread.join(Thread.java:1355)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:952)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:690)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:879)
17/08/31 01:34:26 INFO mapreduce.JobSubmitter: number of splits:4
17/08/31 01:34:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504153900489_0005
17/08/31 01:34:29 INFO impl.YarnClientImpl: Submitted application application_1504153900489_0005
17/08/31 01:34:29 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1504153900489_0005/
17/08/31 01:34:29 INFO mapreduce.Job: Running job: job_1504153900489_0005
17/08/31 01:34:52 INFO mapreduce.Job: Job job_1504153900489_0005 running in uber mode : false
17/08/31 01:34:52 INFO mapreduce.Job:  map 0% reduce 0%
17/08/31 01:37:25 INFO mapreduce.Job:  map 50% reduce 0%
17/08/31 01:38:48 INFO mapreduce.Job:  map 100% reduce 0%
17/08/31 01:38:50 INFO mapreduce.Job: Job job_1504153900489_0005 completed successfully
17/08/31 01:38:51 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=609708
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=409
        HDFS: Number of bytes written=158
        HDFS: Number of read operations=16
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=8
    Job Counters 
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=234310144
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=457637
        Total vcore-milliseconds taken by all map tasks=457637
        Total megabyte-milliseconds taken by all map tasks=234310144
    Map-Reduce Framework
        Map input records=6
        Map output records=6
        Input split bytes=409
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=8848
        CPU time spent (ms)=7840
        Physical memory (bytes) snapshot=445177856
        Virtual memory (bytes) snapshot=2910158848
        Total committed heap usage (bytes)=188219392
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=158
17/08/31 01:38:51 INFO mapreduce.ImportJobBase: Transferred 158 bytes in 271.6628 seconds (0.5816 bytes/sec)
17/08/31 01:38:51 INFO mapreduce.ImportJobBase: Retrieved 6 records.
17/08/31 01:38:51 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader@7d487b8b
17/08/31 01:38:51 DEBUG hive.HiveImport: Hive.inputTable: employee
17/08/31 01:38:51 DEBUG hive.HiveImport: Hive.outputTable: rkdb_hive.employee11
17/08/31 01:38:51 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM `employee` AS t LIMIT 1
17/08/31 01:38:51 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
17/08/31 01:38:52 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
17/08/31 01:38:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
17/08/31 01:38:52 DEBUG manager.SqlManager: Found column id of type [4, 11, 0]
17/08/31 01:38:52 DEBUG manager.SqlManager: Found column name of type [12, 20, 0]
17/08/31 01:38:52 DEBUG manager.SqlManager: Found column dept of type [12, 10, 0]
17/08/31 01:38:52 DEBUG manager.SqlManager: Found column salary of type [4, 10, 0]
17/08/31 01:38:52 DEBUG hive.TableDefWriter: Create statement: CREATE TABLE `rkdb_hive.employee11` ( `id` INT, `name` STRING, `dept` STRING, `salary` INT) COMMENT 'Imported by sqoop on 2017/08/31 01:38:52' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054' LINES TERMINATED BY '\012' STORED AS TEXTFILE
17/08/31 01:38:52 DEBUG hive.TableDefWriter: Load statement: LOAD DATA INPATH 'hdfs://quickstart.cloudera:8020/user/cloudera/ImportSqoop12' INTO TABLE `rkdb_hive.employee11`
17/08/31 01:38:52 INFO hive.HiveImport: Loading uploaded data into Hive
17/08/31 01:38:52 DEBUG hive.HiveImport: Using in-process Hive instance.
17/08/31 01:38:52 DEBUG util.SubprocessSecurityManager: Installing subprocess security manager

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.12.0.jar!/hive-log4j.properties
hadoopHivesqoopsqoop2

2条答案

按热度按时间
yduiuuwa

yduiuuwa1#

请包括 import 以及 --hive-import 在sqoop导入命令中。

赞(0）分享  回复(0）举报  2021-05-29
ie3xauqp

ie3xauqp2#

您的sqoop作业已成功完成。在将数据加载到依赖于zookeeper的hive时，任务被卡住了。只需从cloudera管理器手动打开zookeeper服务。

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com