nosuchmethoderror

v1uwarro  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(202)

我正在尝试使用amazon的emr服务在现场示例上运行mapreduce作业。其目的是从s3读取文件,在mr作业中处理它们,并将行发送到reducer中的cassandra db。
我的自定义jar在我的机器上托管的单节点hadoop集群上运行良好,从s3读取文件并将行发送到云托管的cassandra集群。
在emr spot示例上运行时,我的作业完成其Map阶段而没有问题,然后在reduce阶段抛出以下错误:

2013-07-09 17:23:46,598 FATAL org.apache.hadoop.mapred.Child (main): Error running child : java.lang.NoSuchMethodError: org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.<init>(IIIIIZ)V
    at org.apache.cassandra.transport.Frame$Decoder.<init>(Frame.java:147)
    at com.datastax.driver.core.Connection$PipelineFactory.getPipeline(Connection.java:616)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:212)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
    at com.datastax.driver.core.Connection.<init>(Connection.java:111)
    at com.datastax.driver.core.Connection.<init>(Connection.java:56)
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:387)
    at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:211)
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:174)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:87)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:609)
    at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:553)
    at com.datastax.driver.core.Cluster.<init>(Cluster.java:67)
    at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:94)
    at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:534)
    at com.fullcontact.photo_deduplication.metadata.MetaToCassandraReducer.setup(MetaToCassandraReducer.java:32)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:663)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:426)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

gradle报告说,我唯一的netty依赖是3.6.3版本,它确实有一个有问题的ctor版本。
我的机器和emr之间有什么不同会导致此错误?或者完全是别的原因?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题