添加外部jar时hadoop noclassdeffounderror

cedebl8k  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(470)

我正在尝试使用外部jar在hadoop上运行mapreduce作业。我已经使用 copyFromLocal 至 /user/hduser/lib/ .
在我的main方法中，我将jar添加到distributedcache。但是，当我运行mapreduce程序时，在mapper类中收到一个noclassdeffounderror。我已经尝试了许多解决方案张贴在所以从其他人谁有一个类似的错误，但我没有解决这个问题。感谢您的指导。
从主方法：

Configuration conf = new Configuration();
String jarToAdd1 = "/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar";
String jarToAdd2 = "/user/hduser/lib/joda-time-2.9.1.jar";
addJarToDistributedCache(jarToAdd1, conf);
addJarToDistributedCache(jarToAdd2, conf);
.........

添加到分布式缓存：

private static void addJarToDistributedCache(String jarToAdd, Configuration conf) throws IOException {

        Path hdfsJar = new Path(jarToAdd);
        DistributedCache.addFileToClassPath(hdfsJar,conf);
    }

发生错误的Map器：

public static class Map1 extends Mapper<LongWritable, Text, IntWritable, UserData> {

Map<IntWritable, UserData> userLog = new HashMap<IntWritable, UserData>();

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

    String line = value.toString();                             
    StringTokenizer tokenizer = new StringTokenizer(line);      
    DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyyddmm HH:mm:ss");    //*********ERROR HAPPENS HERE**********

堆栈跟踪：

16/01/30 20:45:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/30 20:45:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/01/30 20:45:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/01/30 20:45:16 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
16/01/30 20:45:16 INFO mapreduce.JobSubmitter: number of splits:3
16/01/30 20:45:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1787244132_0001
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar as file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1.jar as file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/01/30 20:45:17 INFO mapreduce.Job: Running job: job_local1787244132_0001
16/01/30 20:45:17 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/01/30 20:45:18 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Waiting for map tasks
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000000_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/sentimentFeedback7.csv:0+143748596
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000001_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:0+134217728
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000002_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:134217728+105486421
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: map task executor complete.
16/01/30 20:45:18 WARN mapred.LocalJobRunner: job_local1787244132_0001
java.lang.Exception: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:45)
    at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java:55)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 12 more
16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 running in uber mode : false
16/01/30 20:45:18 INFO mapreduce.Job:  map 0% reduce 0%
16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 failed with state FAILED due to: NA
16/01/30 20:45:19 INFO mapreduce.Job: Counters: 0
JavahadoopJardistributed-cacheNoClassDefFoundError

1条答案

按热度按时间
c7rzv4ha

c7rzv4ha1#

首先distributedcache.addfiletoclasspath现在已不推荐使用，但您可以使用：

Configuration conf = getConf();
    Job job = Job.getInstance(conf, "My Job Name");
    job.addFileToClassPath(new Path("path"));

如果您仍然面临同样的问题，请手动执行，我的意思是通过将jar文件放入hadoop的类路径来将代码部署到hadoop本身：
1） 使用以下方法将需要执行的类导出到hadoop中：
在项目资源管理器中，右键单击项目名称并选择“导出…”
在“java”部分下，选择“jar文件”，然后选择“下一步”
选择要将导出的“jar”存储到的目录
2） 转到运行hadoop的服务器并运行以下命令以了解hadoop正在查找的目录，其中一个示例是“$hadoop\u home/share/hadoop/common/lib/*”

$ hadoop classpath

3） 将导出的jar文件从步骤1复制到步骤2中的一个目录，然后再次开始测试。
提示最好直接在服务器中测试map reduce，并使用以下命令在服务器中运行它：

$ hadoop jar JarFileName.jar PakageName.ClassName /path/to/input /path/to/output

有关详细说明的完整示例，请检查此项

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com