运行emr示例,获取301错误

xwbd5t1u  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(417)

我正在尝试运行示例hadoop流命令:

hadoop-streaming -files streamingCode/wordSplitter.py \
-mapper wordSplitter.py \
-input s3://elasticmapreduce/samples/wordcount/input \
-output streamingCode/wordCountOut \
-reducer aggregate

但我一直有个错误:

Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 98038E504E150CEC), S3 Extended Request ID: IW1x5otBSepAnPgW/RKELCUI9dhADQvrXqU2Ase1CLIa0SWDFnBbTscXihrvHvNm2ZRxjjSJZ1Q=

我想这是因为我所在的集群 us-west-2 ,但我不知道如何正确格式化 s3 url(或者这根本不是问题所在)。
编辑:将其更改为以下url后:

s3://s3-us-west-2.amazonaws.com/elasticmapreduce/samples/wordcount/input

我现在得到以下错误:

Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3
Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: BC8DB415C780DF84), 
           S3 Extended Request ID: sx8W/+gvND2ssqQce9ZQsZTiqxmSJYZs8OiXgrjwL3dm0JRPaC7ceHor+yrHsPuKTjM2LUwkRAw=

编辑:所以我确认这个错误确实是因为我的集群在运行 us-west-2 ,我在中创建了一个群集 us-east-1 而且工作正常。所以,问题是如何从另一个区域访问s3 bucket?这有可能吗?

sulc1iza

sulc1iza1#

amazon从emr-4.7.0开始更改了默认行为,这在我们升级emr版本时导致了这个错误。
解决方案很简单,将此配置添加到核心站点:fs.s3n.endpoint=s3.amazonaws.com

相关问题