hadoop2.6.x和amazonaws-sdk库冲突http-core冲突

ifmq2ha2 于 2021-06-02 发布在 Hadoop

关注(0)|答案(1)|浏览(364)

我的工作是将每条记录写入hadoopMap中的dynamodb。
我不能让它用hadoop2.6运行，它有httpclient-4.2.5.jar和httpcore-4.2.5.jar。
我使用的aws是使用httpclient-4.5.2.jar和httpcore-4.4.4.jar构建的。
当我使用classpath包含新的jar文件时，它给出了以下异常。

java.lang.Exception: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:87)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:49)

在我看来，hadoop似乎是使用旧的库构建的，语法也发生了一些变化。
有什么比重新编译旧的aws源更合理的解决方案呢？

Java hadoop amazon-web-services

来源：https://stackoverflow.com/questions/37654549/hadoop-2-6-x-and-amazon-aws-sdk-library-conflicts-http-core-conflict

1条答案

按热度按时间

pes8fvy91#

作为一个更新，我不得不切换到maven并对版本进行一些修改。

<!-- http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.6.0</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.6.0</version>
</dependency>

 <!-- http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
 <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.3.4</version>
      <scope>compile</scope>
    </dependency>

<dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-dynamodb</artifactId>
        <version>1.9.13</version>
    </dependency>

    <dependency>
        <groupId>org.netpreserve.commons</groupId>
        <artifactId>webarchive-commons</artifactId>
        <version>1.1.4</version>
    </dependency>

最后，它成功了

赞(0）回复(0）举报 2021-06-02

我来回答

hadoop2.6.x和amazonaws-sdk库冲突http-core冲突

1条答案

相关问题

热门标签

最新问答