我正在尝试让hadoop3.1.0在windows10系统上运行。
我收到的例外情况是:
2019-11-19 14:49:13,310 INFO mapreduce.Job: map 100% reduce 0%
2019-11-19 14:49:13,325 INFO mapred.LocalJobRunner: reduce task executor complete.
2019-11-19 14:49:13,435 WARN mapred.LocalJobRunner: job_local1214244919_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
**Caused by: java.io.FileNotFoundException: File C:/tmp/hadoop-king%20lui/mapred/local/localRunner/king%20lui/jobcache/job_local1214244919_0001/attempt_local1214244919_0001_m_000001_0/output/file.out.index does not exist**
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:211)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:950)
at org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:152)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125)
at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
**2019-11-19 14:49:14,357 INFO mapreduce.Job: Job job_local1214244919_0001 failed with state FAILED due to: NA**
2019-11-19 14:49:14,372 INFO mapreduce.Job: Counters: 18
File System Counters
FILE: Number of bytes read=840
FILE: Number of bytes written=991668
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=2
Map output records=8
Map output bytes=82
Map output materialized bytes=85
Input split bytes=204
Combine input records=8
Combine output records=6
Spilled Records=6
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=577241088
File Input Format Counters
Bytes Read=48
(我用**强调了我认为重要的部分)
我正在运行hadoop的介绍源代码。
缺少文件的文件夹已存在。
我删除了整个tmp文件夹->没有帮助。
我试过mapreduce不产生输出,但是没有这样的条目。加一个也没用。
2条答案
按热度按时间kadbb4591#
您需要以用户的身份运行hadoop和代码,名称中不带空格(不带空格)
king lui
)测试这种情况的一种方法是
然后可能会错误地说/user/myuser不存在,但您可以创建该目录
r6l8ljro2#
将用户名改为不带空格的有效。
我以前试过这个
set HADOOP_USER_NAME=myuser
,所以我不知道这是否有用。