为了让spark（spark-2.3.0-bin-without-hadoop）在hdfs上使用yarn，我将hadoop降级为hadoop-2.7.6，以解决依赖性问题。
到目前为止，hdfs和Yarn都没有问题。
当我提交一个spark jar时，它崩溃了，我得到了以下stacktrace：

Exception in thread "main" java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
    at org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics(NettyMemoryMetrics.java:80)
    at org.apache.spark.network.util.NettyMemoryMetrics.<init>(NettyMemoryMetrics.java:76)
    at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:109)
    at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
    at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:515)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:800)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
    at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:799)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:824)
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>

当我启动自己的程序和spark提供的示例时，都会发生这种情况。我很确定我在spark-env.sh中正确设置了类路径：

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath):/usr/local/spark/jars/*

生成的类路径如下所示：

SPARK_DIST_CLASSPATH='/usr/local/hadoop/etc/hadoop:/usr/local/hadoop//share/hadoop/common/lib/*:/usr/local/hadoop//share/hadoop/common/*:/usr/local/hadoop//share/hadoop/hdfs:/usr/local/hadoop//share/hadoop/hdfs/lib/*:/usr/local/hadoop//share/hadoop/hdfs/*:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.6/share/hadoop/yarn/*:/usr/local/hadoop//share/hadoop/mapreduce/lib/*:/usr/local/hadoop//share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/usr/local/spark/jars/*'

我不知道怎么解决这个问题。除了配置问题之外，我最好的猜测是另一个库不兼容。在这种情况下，有没有人能告诉我一个spark/hadoop的组合，它实际上没有冲突？

find . -name netty*
./spark-2.3.0-bin-without-hadoop/jars/netty-3.9.9.Final.jar
./spark-2.3.0-bin-without-hadoop/jars/netty-all-4.1.17.Final.jar
./hadoop-2.7.6/share/hadoop/yarn/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/tools/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
./hadoop-2.7.6/share/hadoop/common/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/netty-all-4.0.23.Final.jar
./hadoop-2.7.6/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar