hadoop:reduce阶段开始:filenotfoundexception output/file.out.index不存在

bttbmeg0  于 2021-05-27  发布在  Hadoop
关注(0)|答案(2)|浏览(479)

我正在尝试让hadoop3.1.0在windows10系统上运行。
我收到的例外情况是:

2019-11-19 14:49:13,310 INFO mapreduce.Job:  map 100% reduce 0%
2019-11-19 14:49:13,325 INFO mapred.LocalJobRunner: reduce task executor complete.
2019-11-19 14:49:13,435 WARN mapred.LocalJobRunner: job_local1214244919_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1
        at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

**Caused by: java.io.FileNotFoundException: File C:/tmp/hadoop-king%20lui/mapred/local/localRunner/king%20lui/jobcache/job_local1214244919_0001/attempt_local1214244919_0001_m_000001_0/output/file.out.index does not exist**

        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
        at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:211)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:950)
        at org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:152)
        at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
        at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
        at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
        at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125)
        at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
        at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)

**2019-11-19 14:49:14,357 INFO mapreduce.Job: Job job_local1214244919_0001 failed with state FAILED due to: NA**

2019-11-19 14:49:14,372 INFO mapreduce.Job: Counters: 18
        File System Counters
                FILE: Number of bytes read=840
                FILE: Number of bytes written=991668
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
        Map-Reduce Framework
                Map input records=2
                Map output records=8
                Map output bytes=82
                Map output materialized bytes=85
                Input split bytes=204
                Combine input records=8
                Combine output records=6
                Spilled Records=6
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=0
                Total committed heap usage (bytes)=577241088
        File Input Format Counters
                Bytes Read=48

(我用**强调了我认为重要的部分)
我正在运行hadoop的介绍源代码。
缺少文件的文件夹已存在。
我删除了整个tmp文件夹->没有帮助。
我试过mapreduce不产生输出,但是没有这样的条目。加一个也没用。

kadbb459

kadbb4591#

您需要以用户的身份运行hadoop和代码,名称中不带空格(不带空格) king lui )
测试这种情况的一种方法是

set HADOOP_USER_NAME=myuser 
yarn jar your-app.jar

然后可能会错误地说/user/myuser不存在,但您可以创建该目录

r6l8ljro

r6l8ljro2#

将用户名改为不带空格的有效。
我以前试过这个 set HADOOP_USER_NAME=myuser ,所以我不知道这是否有用。

相关问题