words count输出显示mapred而不是mapreduce

mrwjdhj3  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(213)

我刚刚将ubuntu13.10配置为在伪分布式模式下进行mapreduce代码开发。我已经安装了hadoop0.20.2版本的hadoop。一切都很顺利,我也可以开始所有五个执事。
在同一台机器上,我下载了eclipse并向其中添加了所有基于hadoop的库。我可以直接从eclipseide运行map reduce word count示例。唯一困扰我的是,当我运行我的字数计算示例时,它在控制台中打印如下内容:

13/09/23 16:11:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your  
platform... using builtin-java classes where applicable
13/09/23 16:11:05 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See    
JobConf(Class) or JobConf#setJar(String).
13/09/23 16:11:05 INFO input.FileInputFormat: Total input paths to process : 1
13/09/23 16:11:06 INFO mapred.JobClient: Running job: job_local_0001
13/09/23 16:11:06 INFO util.ProcessTree: setsid exited with exit code 0
13/09/23 16:11:06 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c931fc
13/09/23 16:11:06 INFO mapred.MapTask: io.sort.mb = 100
13/09/23 16:11:07 INFO mapred.JobClient:  map 0% reduce 0%
13/09/23 16:11:07 INFO mapred.MapTask: data buffer = 79691776/99614720
13/09/23 16:11:07 INFO mapred.MapTask: record buffer = 262144/327680
13/09/23 16:11:08 INFO mapred.MapTask: Starting flush of map output
13/09/23 16:11:08 INFO mapred.MapTask: Finished spill 0
13/09/23 16:11:08 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the 
process of commiting
13/09/23 16:11:09 INFO mapred.LocalJobRunner: 
13/09/23 16:11:09 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
13/09/23 16:11:09 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1342ba4
13/09/23 16:11:09 INFO mapred.LocalJobRunner: 
13/09/23 16:11:09 INFO mapred.Merger: Merging 1 sorted segments
13/09/23 16:11:10 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total 
size: 48 bytes
13/09/23 16:11:10 INFO mapred.LocalJobRunner: 
13/09/23 16:11:10 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the 
process of commiting
13/09/23 16:11:10 INFO mapred.LocalJobRunner: 
13/09/23 16:11:10 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
13/09/23 16:11:10 INFO output.FileOutputCommitter: Saved output of task 
'attempt_local_0001_r_000000_0' to outputWords
13/09/23 16:11:10 INFO mapred.JobClient:  map 100% reduce 0%
13/09/23 16:11:12 INFO mapred.LocalJobRunner: reduce > reduce
13/09/23 16:11:12 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
13/09/23 16:11:12 WARN mapred.LocalJobRunner: job_local_0001

java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:284)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
 Exception in thread "Thread-1" java.lang.NoClassDefFoundError:    
org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:300)
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
13/09/23 16:11:13 INFO mapred.JobClient:  map 100% reduce 100%
13/09/23 16:11:13 INFO mapred.JobClient: Job complete: job_local_0001
13/09/23 16:11:13 INFO mapred.JobClient: Counters: 20
13/09/23 16:11:13 INFO mapred.JobClient:   File Output Format Counters 
13/09/23 16:11:13 INFO mapred.JobClient:     Bytes Written=42
13/09/23 16:11:13 INFO mapred.JobClient:   FileSystemCounters
13/09/23 16:11:13 INFO mapred.JobClient:     FILE_BYTES_READ=534
13/09/23 16:11:13 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=63640
13/09/23 16:11:13 INFO mapred.JobClient:   File Input Format Counters 
13/09/23 16:11:13 INFO mapred.JobClient:     Bytes Read=63
13/09/23 16:11:13 INFO mapred.JobClient:   Map-Reduce Framework
13/09/23 16:11:13 INFO mapred.JobClient:     Map output materialized bytes=52
13/09/23 16:11:13 INFO mapred.JobClient:     Map input records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce shuffle bytes=0
13/09/23 16:11:13 INFO mapred.JobClient:     Spilled Records=8
13/09/23 16:11:13 INFO mapred.JobClient:     Map output bytes=110
13/09/23 16:11:13 INFO mapred.JobClient:     Total committed heap usage (bytes)=231350272
13/09/23 16:11:13 INFO mapred.JobClient:     CPU time spent (ms)=0
13/09/23 16:11:13 INFO mapred.JobClient:     SPLIT_RAW_BYTES=124
13/09/23 16:11:13 INFO mapred.JobClient:     Combine input records=12
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce input records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce input groups=4
13/09/23 16:11:13 INFO mapred.JobClient:     Combine output records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce output records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient:     Map output records=12

在以上输出中,如果您看到有几件事我不确定是否正确:
它的printing mapred.jobclient:mapred是hadoop的旧库,所以我如何将它添加到mapreduce(alreday在eclipse中添加了最近的新库,仍然得到相同的mapred消息)
为什么会出现这个错误:java.lang.noclassdeffounderror
我能够得到输出目录生成正确的结果也。
如果需要任何其他细节,请告诉我。
希望得到答案。
hadooping快乐!!!

rqdpfwrv

rqdpfwrv1#

你越来越--

Exception in thread "Thread-1" java.lang.NoClassDefFoundError:    
org/apache/commons/httpclient/HttpMethod

因为,在类路径中没有包含一些依赖jar。
请尝试在lib/目录中包含以下内容,然后重试--

commons-httpclient-3.1.jar
commons-cli-1.2.jar
commons-logging-1.0.4.jar
commons-logging-api-1.0.4.jar
log4j-1.2.15.jar
commons-cli-1.2.jar
jackson-core-asl-1.5.2.jar
jackson-mapper-asl-1.5.2.jar

如果包含这些不起作用,请包含lib/目录中的所有jar。
此外, mapred.JobClient hadoop(mapred api或mapreduce api)没有弃用和引用。

相关问题