我正在尝试运行示例hadoop流命令:
hadoop-streaming -files streamingCode/wordSplitter.py \
-mapper wordSplitter.py \
-input s3://elasticmapreduce/samples/wordcount/input \
-output streamingCode/wordCountOut \
-reducer aggregate
但我一直有个错误:
Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 98038E504E150CEC), S3 Extended Request ID: IW1x5otBSepAnPgW/RKELCUI9dhADQvrXqU2Ase1CLIa0SWDFnBbTscXihrvHvNm2ZRxjjSJZ1Q=
我想这是因为我所在的集群 us-west-2
,但我不知道如何正确格式化 s3
url(或者这根本不是问题所在)。
编辑:将其更改为以下url后:
s3://s3-us-west-2.amazonaws.com/elasticmapreduce/samples/wordcount/input
我现在得到以下错误:
Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3
Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: BC8DB415C780DF84),
S3 Extended Request ID: sx8W/+gvND2ssqQce9ZQsZTiqxmSJYZs8OiXgrjwL3dm0JRPaC7ceHor+yrHsPuKTjM2LUwkRAw=
编辑:所以我确认这个错误确实是因为我的集群在运行 us-west-2
,我在中创建了一个群集 us-east-1
而且工作正常。所以,问题是如何从另一个区域访问s3 bucket?这有可能吗?
1条答案
按热度按时间sulc1iza1#
amazon从emr-4.7.0开始更改了默认行为,这在我们升级emr版本时导致了这个错误。
解决方案很简单,将此配置添加到核心站点:fs.s3n.endpoint=s3.amazonaws.com