将现有作业从AWS Glue 2.0升级到3.0时,当前scala版本为2.11.8,Spark为3.1
Exception in User Class: java.lang.NoSuchMethodError : scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
stacktrace错误:
ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): Exception in User Class: java.lang.NoSuchMethodError : scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
com.utils.CommonUtils.getPathPrefixFromFilePath(CommonUtils.scala:41)
com.awsutils.S3Service.downloadObject(S3Service.scala:169)
com.ValidateHealthData.getDatesToValidate(ValidateHealthData.scala:67)
com.ValidateHealthData.validateHealthData(ValidateHealthData.scala:46)
com.ValidateHealthData.main(ValidateHealthData.scala:287)
EtlMain$.main(EtlMain.scala:106)
EtlMain.main(EtlMain.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke(ProcessLauncher.scala:48)
com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke$(ProcessLauncher.scala:48)
com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:90)
com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:161)
com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:30)
com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)
您能告诉我如何解决此升级问题吗?
1条答案
按热度按时间kkih6yb81#
在标题为Migrating AWS Glue for Spark jobs to AWS Glue version 3.0的文档中说:
根据你说你使用的scala版本是
2.11
,你应该先升级到2.12
。至少,这应该是你的问题之一。迁移指南解释了如何继续迁移
AWS Glue 3.0
。您有一个名为Actions to migrate to AWS Glue 3.0
的部分和另一个名为Migration check list
的部分。