spark提交作业在客户端模式下获得成功,但在群集模式下失败

wecizke3  于 2021-05-18  发布在  Spark
关注(0)|答案(0)|浏览(404)

使用ansible playbook在spark群集上安装了spark-3.0.1版本,并在运行spark submit的spark作业时添加了spark配置参数(spark\u home、hadoop\u conf\u dir、yarn\u conf\u dir、java\u home):

bash spark-submit --master yarn --deploy-mode cluster --name EmpDataset --driver-memory 3g --executor-memory 10g --executor-cores 3 --num-executors 5 --class <className> <jarPath>.jar

在群集模式下出现以下错误:

INFO spark.SparkContext: Successfully stopped SparkContext
20/11/04 07:09:34 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 28499
    at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:532)
    at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:315)
    at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:102)
    at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:76)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:45)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:59)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:59)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$$Lambda$885/0x000000000ad971d0.apply(Unknown Source)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$Lambda$131/0x0000000009897d70.apply(Unknown Source)
    at scala.collection.Iterator.foreach(Iterator.scala:941)
    at scala.collection.Iterator.foreach$(Iterator.scala:941)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:59)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:181)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$$Lambda$884/0x000000000ad969d0.apply(Unknown Source)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
    at scala.collection.TraversableLike$$Lambda$55/0x000000000974cf30.apply(Unknown Source)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)

如何解决上述问题?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题