我在运行map reduce java应用程序时遇到问题我使用aws提供的教程代码简化了问题,该教程运行预定义步骤:
public class Main {
public static void main(String[] args) {
AWSCredentials credentials = getCredentials();
AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(
credentials);
StepFactory stepFactory = new StepFactory();
StepConfig enabledebugging = new StepConfig()
.withName("Enable debugging")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newEnableDebuggingStep());
StepConfig installHive = new StepConfig().withName("Install Hive")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newInstallHiveStep());
RunJobFlowRequest request = new RunJobFlowRequest()
.withName("Hive Interactive")
.withAmiVersion("3.3.1")
.withSteps(enabledebugging, installHive)
.withLogUri("s3://tweets-hadoop/")
.withServiceRole("service_role")
.withJobFlowRole("jobflow_role")
.withInstances(
new JobFlowInstancesConfig().withEc2KeyName("hadoop")
.withInstanceCount(5)
.withKeepJobFlowAliveWhenNoSteps(true)
.withMasterInstanceType("m3.xlarge")
.withSlaveInstanceType("m1.large"));
RunJobFlowResult result = emr.runJobFlow(request);
System.out.println(result);
}
private static AWSCredentials getCredentials() {
AWSCredentials credentials = null;
credentials = new BasicAWSCredentials("<KEY>","<VALUE>");
return credentials;
}
}
其中,'hadoop'是我在ec2控制台中创建的密钥对。
运行后,我看到作业试图在emr控制台中启动,1分钟后,它从“启动”更改为“终止时出错验证错误”
没有提供其他信息
有什么问题吗?
谢谢!
1条答案
按热度按时间qnyhuwrf1#
您可以在emr群集列表详细信息页(顶部)上查看错误的详细信息。因为这是一个验证错误,它还没有出现在日志中,所以唯一的方法是检查异常详细信息/cli响应/aws控制台。。。
我猜您使用的示例类型在emr中不受支持(它们在ec2中受支持,但在emr中不受支持)。然而,一旦你遵循给定的步骤,你将得到确切的问题。