由于serde2/serde,aws emr配置单元失败

gdrx4gfi  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(427)

我正在s3上运行emr配置单元查询,它显示“map operator initialization failed”失败
我试着 HADOOP_CLASSPATH 如下所示,仍然没有运气。

set HADOOP_CLASSPATH=/usr/lib/hive/lib/*;

另外,在jar下面加上, add jar /usr/hive/json-serde-1.3.7-jar-with-dependencies.jar . 这个jar文件存在于给定的路径中。
我不知道为什么它会抱怨如下:

Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe

日志如下:
状态:failed vertex failed,vertexname=map 10,vertexid=vertex\u 1515176779844\u 0076\u 2\u 02,diagnostics=[任务失败,taskid=task\u 1515176779844\u 0076\u 2\u 02\u 000031,diagnostics=[任务尝试0失败,info=[错误:运行任务时出错(失败):尝试\u 1515176779844 \u 0076 \u 2 \u 02 \u 000031 \u 0:java.lang.runtimeexception:java.lang.runtimeexception:map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:211)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.run(tezprocessor。java:168)位于org.apache.tez.runtime.logicalioprocessorruntimetask.run(logicalioprocessorruntimetask)。java:370)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:73)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:61)在javax.security.auth.subject.doas(主题)中的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1698)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:61)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:37)在org.apache.tez.common.callablewithndc.call(callablewithndc。java:36)在java.util.concurrent.futuretask.run(futuretask。java:266)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:java.lang.runtimeexception:Map运算符初始化在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor)失败。java:354)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:184) ... 还有14个原因:java.lang.noclassdeffounderror:org/apache/hadoop/hive/serde2/serde at java.lang.classloader.defineclass1(本机方法)at java.lang.classloader.defineclass(classloader)。java:763)在java.security.secureclassloader.defineclass(secureclassloader。java:142)在java.net.urlclassloader.defineclass(urlclassloader。java:467)在java.net.urlclassloader.access$100(urlclassloader。java:73)在java.net.urlclassloader$1.run(urlclassloader。java:368)在java.net.urlclassloader$1.run(urlclassloader。java:362)位于java.net.urlclassloader.findclass(urlclassloader)的java.security.accesscontroller.doprivileged(本机方法)。java:361)在java.lang.classloader.loadclass(类加载器。java:424)在sun.misc.launcher$appclassloader.loadclass(launcher。java:335)在java.lang.classloader.loadclass(classloader。java:357)在java.lang.class.forname0(本机方法)在java.lang.class.forname(类。java:348)在org.apache.hadoop.conf.configuration.getclassbynameornull(配置。java:2134) 在org.apache.hadoop.conf.configuration.getclassbyname(配置。java:2099)在org.apache.hadoop.hive.ql.plan.partitiondesc.getdeserializer(partitiondesc。java:177)在org.apache.hadoop.hive.ql.exec.mapoperator.getconvertedoi(mapoperator。java:295)在org.apache.hadoop.hive.ql.exec.mapoperator.setchildren(mapoperator。java:420)在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor。java:286) ... 还有15个原因:java.lang.classnotfoundexception:org.apache.hadoop.hive.serde2.serde at java.net.urlclassloader.findclass(urlclassloader)。java:381)在java.lang.classloader.loadclass(classloader。java:424)在sun.misc.launcher$appclassloader.loadclass(launcher。java:335)在java.lang.classloader.loadclass(classloader。java:357) ... 还有35次,任务尝试1失败,info=[错误:运行任务时出错(失败):尝试\u 1515176779844 \u 0076 \u 2 \u 02 \u 000031 \u 1:java.lang.runtimeexception:java.lang.runtimeexception:map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:211)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.run(tezprocessor。java:168)位于org.apache.tez.runtime.logicalioprocessorruntimetask.run(logicalioprocessorruntimetask)。java:370)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:73)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:61)在javax.security.auth.subject.doas(主题)中的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1698)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:61)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:37)在org.apache.tez.common.callablewithndc.call(callablewithndc。java:36)在java.util.concurrent.futuretask.run(futuretask。java:266)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:java.lang.runtimeexception:Map运算符初始化在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor)失败。java:354)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:184) ... 还有14个原因:java.lang.noclassdeffounderror:org/apache/hadoop/hive/serde2/serde at java.lang.classloader.defineclass1(本机方法)at java.lang.classloader.defineclass(classloader)。java:763)在java.security.secureclassloader.defineclass(secureclassloader。java:142)在java.net.urlclassloader.defineclass(urlclassloader。java:467)在java.net.urlclassloader.access$100(urlclassloader。java:73)在java.net.urlclassloader$1.run(urlclassloader。java:368)在java.net.urlclassloader$1.run(urlclassloader。java:362)位于java.net.urlclassloader.findclass(urlclassloader)的java.security.accesscontroller.doprivileged(本机方法)。java:361)在java.lang.classloader.loadclass(类加载器。java:424)在sun.misc.launcher$appclassloader.loadclass(launcher。java:335)在java.lang.classloader.loadclass(classloader。java:357)在java.lang.class.forname0(本机方法)在java.lang.class.forname(类。java:348)在org.apache.hadoop.conf.configuration.getclassbynameornull(配置。java:2134) 在org.apache.hadoop.conf.configuration.getclassbyname(配置。java:2099)在org.apache.hadoop.hive.ql.plan.partitiondesc.getdeserializer(partitiondesc。java:177)在org.apache.hadoop.hive.ql.exec.mapoperator.getconvertedoi(mapoperator。java:295)在org.apache.hadoop.hive.ql.exec.mapoperator.setchildren(mapoperator。java:420)在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor。java:286) ... 还有15个原因:java.lang.classnotfoundexception:org.apache.hadoop.hive.serde2.serde at java.net.urlclassloader.findclass(urlclassloader)。java:381)在java.lang.classloader.loadclass(classloader。java:424)在sun.misc.launcher$appclassloader.loadclass(launcher。java:335)在java.lang.classloader.loadclass(classloader。java:357) ... 还有35次,任务尝试2失败,info=[错误:运行任务时出错(失败):尝试\u 1515176779844 \u 0076 \u 2 \u 02 \u 000031 \u 2:java.lang.runtimeexception:java.lang.runtimeexception:map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:211)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.run(tezprocessor。java:168)位于org.apache.tez.runtime.logicalioprocessorruntimetask.run(logicalioprocessorruntimetask)。java:370)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:73)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:61)在javax.security.auth.subject.doas(主题)中的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1698)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:61)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:37)在org.apache.tez.common.callablewithndc.call(callablewithndc。java:36)在java.util.concurrent.futuretask.run(futuretask。java:266)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:java.lang.runtimeexception:Map运算符初始化在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor)失败。java:354)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:184) ... 还有14个原因:java.lang.noclassdeffounderror:org/apache/hadoop/hive/serde2/serde at java.lang.classloader.defineclass1(本机方法)at java.lang.classloader.defineclass(classloader)。java:763)在java.security.secureclassloader.defineclass(secureclassloader。java:142)在java.net.urlclassloader.defineclass(urlclassloader。java:467)在java.net.urlclassloader.access$100(urlclassloader。java:73)在java.net.urlclassloader$1.run(urlclassloader。java:368)在java.net.urlclassloader$1.run(urlclassloader。java:362)位于java.net.urlclassloader.findclass(urlclassloader)的java.security.accesscontroller.doprivileged(本机方法)。java:361)在java.lang.classloader.loadclass(类加载器。java:424)在sun.misc.launcher$appclassloader.loadclass(launcher。java:335)在java.lang.classloader.loadclass(classloader。java:357)在java.lang.class.forname0(本机方法)在java.lang.class.forname(类。java:348)在org.apache.hadoop.conf.configuration.getclassbynameornull(配置。java:2134) 在org.apache.hadoop.conf.configuration.getclassbyname(配置。java:2099)在org.apache.hadoop.hive.ql.plan.partitiondesc.getdeserializer(partitiondesc。java:177)位于org.apache.hadoop.hive.ql.exec.maporator.ge

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题