如何用aws运行map reduce程序?

cnh2zyt3  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(260)

我想用hadoop框架运行一个map reduce程序。我需要在amazonelasticmapreduce示例上运行该程序,但不断出现以下错误:
线程“main”java.lang.nosuchmethoderror中出现异常:com.amazonaws.transform.jsonerrorunmarshaller:method(ljava/lang/class;)未找到v
我试图修复pom文件并添加awssdk,更改其版本并分别添加核心。
我的pom.xml文件:

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.3</version>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.7.3</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-ec2</artifactId>
        <version>1.10.2</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.10.5</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-emr</artifactId>
        <version>1.9.0</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.5</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-core</artifactId>
        <version>1.11.5</version>
    </dependency>

我正在尝试创建示例:

AWSCredentials creds = new PropertiesCredentials(new FileInputStream(propertiesFilePath));

AmazonElasticMapReduce mapReduce = new AmazonElasticMapReduceClient(creds);

我希望程序在aws示例上运行(可能会在我的代码中得到更多的错误,我可以调试)。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题