无法在使用s3存储的ec2中再次运行mapreduce

72qzrwbm  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(348)

我已经在amazonec2示例中设置了单节点hadoop。接下来我可以第一次运行这个示例程序。但是为了让它再次运行,我必须在停止发出stop-all.sh之后删除s3和本地tmp目录中的所有目录和文件。我只运行mapred(tasktracker和jobtracker)。再次尝试重新运行该示例时,我收到错误消息。

hduser@ip-10-252-196-143:~$ hadoop jar ./hadoop/hadoop-examples-1.2.1.jar  wordcount input output2
13/09/20 09:43:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.IOException: The ownership on the staging directory s3://vikesh-hadoop-bucket/home/hduser/tmp/mapred/staging/hduser/.staging is not as expected. It is owned by . The directory must be owned by the submitter hduser or by hduser
java.io.IOException: The ownership on the staging directory s3://vikesh-hadoop-bucket/home/hduser/tmp/mapred/staging/hduser/.staging is not as expected. It is owned by . The directory must be owned by the submitter hduser or by hduser
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

似乎这是相关的,但我相信是使用cloudera的发行版。
为什么会这样?我能做些什么来解决这个问题?我在64位虚拟机上使用hadoop-1.2.1和openjdk7。
谢谢

nqwrtyyt

nqwrtyyt1#

我们面临同样的问题。我们只需删除每个mapreduce之间的s3://{bucket name}/home/{user}/tmp/mapred/staging/{user}/.staging。。。但如果有人有更好的解决办法。。。

相关问题