Apache Spark 将AWS Glue从2.0升级到3.0时出错

cwtwac6a  于 2023-06-24  发布在  Apache
关注(0)|答案(1)|浏览(169)

将现有作业从AWS Glue 2.0升级到3.0时,当前scala版本为2.11.8,Spark为3.1

Exception in User Class: java.lang.NoSuchMethodError : scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

stacktrace错误:

ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): Exception in User Class: java.lang.NoSuchMethodError : scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
com.utils.CommonUtils.getPathPrefixFromFilePath(CommonUtils.scala:41)
com.awsutils.S3Service.downloadObject(S3Service.scala:169)
com.ValidateHealthData.getDatesToValidate(ValidateHealthData.scala:67)
com.ValidateHealthData.validateHealthData(ValidateHealthData.scala:46)
com.ValidateHealthData.main(ValidateHealthData.scala:287)
EtlMain$.main(EtlMain.scala:106)
EtlMain.main(EtlMain.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke(ProcessLauncher.scala:48)
com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke$(ProcessLauncher.scala:48)
com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:90)
com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:161)
com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:30)
com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)

您能告诉我如何解决此升级问题吗?

kkih6yb8

kkih6yb81#

在标题为Migrating AWS Glue for Spark jobs to AWS Glue version 3.0的文档中说:

  • 你的工作依赖于Scala 2.11吗?
  • AWS Glue 3.0使用Scala 2.12,因此如果您的库依赖于Scala 2.11,则需要使用Scala 2.12重建库。

根据你说你使用的scala版本是2.11,你应该先升级到2.12。至少,这应该是你的问题之一。
迁移指南解释了如何继续迁移AWS Glue 3.0。您有一个名为Actions to migrate to AWS Glue 3.0的部分和另一个名为Migration check list的部分。

相关问题