我正在运行我的第一个mapreduce程序，输出文件似乎有一些权限问题

dwthyt8l  于 2021-05-31  发布在  Hadoop
关注(0)|答案(3)|浏览(283)

[mykishore231087@ip-172-31-20-58 ~]$hadoop-jar/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar
-input mayank/data/upx/wc\u data.txt -output/mayank/output/res.txt -file/home/mykishore231087/d/wordcount\u mapper.py -file/home/mykishore231087/d/wordcount\u reducer.py -mapper“/home/mykishore231087/d/wordcount\u reducer.py”警告：使用“yarn jar”启动Yarn应用程序。18/04/25 21:28:18 warn streaming.streamjob:-文件选项已弃用，请改用常规选项-文件。
packagejobjar:[/home/mykishore231087/d/wordcount\u mapper.py，/home/mykishore231087/d/wordcount\u reducer.py][/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.4.0-3485.jar]/tmp/streamjob8387910308600265451.jar tmpdir=null 18/04/25 21:28:20 info impl.timelineclientimpl:timeline服务地址：http://ip-172-31-13-154.ec2.internal：8188/ws/v1/timeline/18/04/25 21:28:20 info client.rmproxy:正在连接到resourcemanager，ip-172-31-53-48.ec2.内部/172.31.53.48:8050 18/04/25 21:28:21 info impl.timelineclientimpl:时间线服务地址：http://ip-172-31-13-154.ec2.internal：8188/ws/v1/timeline/18/04/25 21:28:21 info client.rmproxy:通过ip-172-31-53-48连接到resourcemanager.ec2.内部/172.31.53.48:8050 18/04/25 21:28:22info mapred.fileinputformat:要处理的总输入路径：1 18/04/25 21:28:22 info mapreduce.jobsubmitter:个splits:2 18/04/25 21:28:23 info mapreduce.jobsubmitter:提交作业令牌：job\u 1524162637175\u 1044 18/04/25 21:28:23 info impl.yarclientimpl:提交的应用程序申请\u 1524162637175\u 1044 18/04/25 21:28:23 infomapreduce.job:跟踪作业的url:http://a.cloudxlab.com：8088/proxy/application\u 1524162637175\u 1044/18/04/25 21:28:23 info mapreduce.job:running job:job\u 1524162637175\u 1044 18/04/25 21:28:32 info mapreduce.job:job job\u 1524162637175\u 1044 running in uber mode:false 18/04/25 21:28:32 info mapreduce.job:map 0%reduce 0%18/04/25 21:28:33info mapreduce.job:作业作业\u 1524162637175 \u 1044失败，状态为失败，原因：作业安装失败：org.apache.hadoop.security.accesscontrolion:权限被拒绝：user=mykishore231087，access=write，inode=“/mayank/output/res.txt/\u temporary/1”：hdfs:hdfs：drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3020)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2988)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1057)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1053)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1053)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1046)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:343)
at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:131)  at org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:265)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mykishore231087, access=WRITE, inode=" /mayank/output/res.txt/_temporary/1":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:558)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy10.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3018)
... 15 more 18/04/25 21:28:33 INFO mapreduce.Job: Counters: 2
Job Counters 
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0 18/04/25 21:28:33 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!
hadoophadoop2

3条答案

按热度按时间
imzjd6km

imzjd6km1#

user=mykishore231087，access=write，inode=“/mayank/output/res.txt/\u temporary/1”：hdfs:hdfs：drwxr-xr-x
您创建了 /mayank 目录作为hdfs超级用户帐户。所有子目录都继承该目录的权限。正如错误所说，文件夹是由hdfs用户独占的，其他所有人都是只读的
您应该创建自己的用户文件夹

sudo su - hdfs 
hadoop fs -mkdir -p /user/mykishore231087
hadoop fs -chmod -R 750 /user/mykishore231087
hadoop fs chown mykishore231087:hdfs  /user/mykishore231087

以该目录作为绝对路径编辑mapreduce代码。。。 /user/mykishore231087/mayank/ 或者，两者之间存在显著差异 output /mayank/ 以及 output mayank/ ，第一个指向hdfs的根目录，另一个是您的用户文件夹（从中读取输入）

赞(0）分享  回复(0）举报  2021-06-01
xtupzzrd

xtupzzrd2#

作为一个hdfs管理用户，您需要运行下面的命令来解决您的问题。hdfs dfs-chown-r mykishore231087:hdfs/user/mykishore231087
这里的问题是您的id没有将数据写入目标目录的有效权限。您可以按降序提供对目标目录的写访问，也可以使用上述命令。inode=“/mayank/output/res.txt/\u temporary/1”：hdfs:hdfs：drwxr**-**x

赞(0）分享  回复(0）举报  2021-06-01
mwngjboj

mwngjboj3#

上面已经给出了答案。只是一个小小的评论hadoop的命令被删除了。使用“yarn”代替日志中提到的“警告：使用“yarn jar”启动yarn应用程序。18/04/25 21:28:18 warn streaming.streamjob:-文件选项已弃用，请改用常规选项-文件。“

赞(0）分享  回复(0）举报  2021-05-31
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com