在mapreduce作业中找不到hadoop类

snvhrwxg  于 2021-05-30  发布在  Hadoop
关注(0)|答案(0)|浏览(264)

我有一个mapreduce的工作,需要一个avro文件作为输入。我将它与所有必需的库(jar库文件)一起导出到一个jar文件中。我有两个不同的集群,一个是hdinsight模拟器,另一个在hdp沙盒中。它在hdp沙盒上运行良好,但在hdinsight模拟器上出现错误,无法找到avroinputformat类。我试着用-libjar选项运行这个作业,但是没有用。以下是错误消息:

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapred.AvroInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1927)
    at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:686)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapred.AvroInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1919)
    ... 9 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.avro.mapred.AvroInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
    ... 10 more

这看起来很奇怪,因为它在一个集群上运行良好!有人知道问题出在哪里吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题