我刚刚将ubuntu13.10配置为在伪分布式模式下进行mapreduce代码开发。我已经安装了hadoop0.20.2版本的hadoop。一切都很顺利,我也可以开始所有五个执事。
在同一台机器上,我下载了eclipse并向其中添加了所有基于hadoop的库。我可以直接从eclipseide运行map reduce word count示例。唯一困扰我的是,当我运行我的字数计算示例时,它在控制台中打印如下内容:
13/09/23 16:11:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
13/09/23 16:11:05 WARN mapred.JobClient: No job jar file set. User classes may not be found. See
JobConf(Class) or JobConf#setJar(String).
13/09/23 16:11:05 INFO input.FileInputFormat: Total input paths to process : 1
13/09/23 16:11:06 INFO mapred.JobClient: Running job: job_local_0001
13/09/23 16:11:06 INFO util.ProcessTree: setsid exited with exit code 0
13/09/23 16:11:06 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c931fc
13/09/23 16:11:06 INFO mapred.MapTask: io.sort.mb = 100
13/09/23 16:11:07 INFO mapred.JobClient: map 0% reduce 0%
13/09/23 16:11:07 INFO mapred.MapTask: data buffer = 79691776/99614720
13/09/23 16:11:07 INFO mapred.MapTask: record buffer = 262144/327680
13/09/23 16:11:08 INFO mapred.MapTask: Starting flush of map output
13/09/23 16:11:08 INFO mapred.MapTask: Finished spill 0
13/09/23 16:11:08 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the
process of commiting
13/09/23 16:11:09 INFO mapred.LocalJobRunner:
13/09/23 16:11:09 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
13/09/23 16:11:09 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1342ba4
13/09/23 16:11:09 INFO mapred.LocalJobRunner:
13/09/23 16:11:09 INFO mapred.Merger: Merging 1 sorted segments
13/09/23 16:11:10 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total
size: 48 bytes
13/09/23 16:11:10 INFO mapred.LocalJobRunner:
13/09/23 16:11:10 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the
process of commiting
13/09/23 16:11:10 INFO mapred.LocalJobRunner:
13/09/23 16:11:10 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
13/09/23 16:11:10 INFO output.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_000000_0' to outputWords
13/09/23 16:11:10 INFO mapred.JobClient: map 100% reduce 0%
13/09/23 16:11:12 INFO mapred.LocalJobRunner: reduce > reduce
13/09/23 16:11:12 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
13/09/23 16:11:12 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:284)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
Exception in thread "Thread-1" java.lang.NoClassDefFoundError:
org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:300)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
13/09/23 16:11:13 INFO mapred.JobClient: map 100% reduce 100%
13/09/23 16:11:13 INFO mapred.JobClient: Job complete: job_local_0001
13/09/23 16:11:13 INFO mapred.JobClient: Counters: 20
13/09/23 16:11:13 INFO mapred.JobClient: File Output Format Counters
13/09/23 16:11:13 INFO mapred.JobClient: Bytes Written=42
13/09/23 16:11:13 INFO mapred.JobClient: FileSystemCounters
13/09/23 16:11:13 INFO mapred.JobClient: FILE_BYTES_READ=534
13/09/23 16:11:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=63640
13/09/23 16:11:13 INFO mapred.JobClient: File Input Format Counters
13/09/23 16:11:13 INFO mapred.JobClient: Bytes Read=63
13/09/23 16:11:13 INFO mapred.JobClient: Map-Reduce Framework
13/09/23 16:11:13 INFO mapred.JobClient: Map output materialized bytes=52
13/09/23 16:11:13 INFO mapred.JobClient: Map input records=4
13/09/23 16:11:13 INFO mapred.JobClient: Reduce shuffle bytes=0
13/09/23 16:11:13 INFO mapred.JobClient: Spilled Records=8
13/09/23 16:11:13 INFO mapred.JobClient: Map output bytes=110
13/09/23 16:11:13 INFO mapred.JobClient: Total committed heap usage (bytes)=231350272
13/09/23 16:11:13 INFO mapred.JobClient: CPU time spent (ms)=0
13/09/23 16:11:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=124
13/09/23 16:11:13 INFO mapred.JobClient: Combine input records=12
13/09/23 16:11:13 INFO mapred.JobClient: Reduce input records=4
13/09/23 16:11:13 INFO mapred.JobClient: Reduce input groups=4
13/09/23 16:11:13 INFO mapred.JobClient: Combine output records=4
13/09/23 16:11:13 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient: Reduce output records=4
13/09/23 16:11:13 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient: Map output records=12
在以上输出中,如果您看到有几件事我不确定是否正确:
它的printing mapred.jobclient:mapred是hadoop的旧库,所以我如何将它添加到mapreduce(alreday在eclipse中添加了最近的新库,仍然得到相同的mapred消息)
为什么会出现这个错误:java.lang.noclassdeffounderror
我能够得到输出目录生成正确的结果也。
如果需要任何其他细节,请告诉我。
希望得到答案。
hadooping快乐!!!
1条答案
按热度按时间rqdpfwrv1#
你越来越--
因为,在类路径中没有包含一些依赖jar。
请尝试在lib/目录中包含以下内容,然后重试--
如果包含这些不起作用,请包含lib/目录中的所有jar。
此外,
mapred.JobClient
hadoop(mapred api或mapreduce api)没有弃用和引用。