我试图做一个aws胶水Spark作业,从一个表读取一些数据到S3位置。但我得到了一个错误,因为下面的堆栈跟踪为“一个错误发生时,调用z:com.amazonaws.services.glue.util.Job.init”。有人能建议一种方法来调试这个问题。
Traceback (most recent call last):
File "C:\opt\inbound_db.py", line 31, in <module>
glue_spark.job.init(
File "C:\opt\Anaconda3\Lib\site-packages\awsglue\job.py", line 42, in init
self._job.init(job_name, self._glue_context._glue_scala_context, args)
File "C:\opt\Anaconda3\Lib\site-packages\py4j\java_gateway.py", line 1322, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "C:\opt\Anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py", line 169, in deco
return f(*a, **kw)
^^^^^^^^^^^
File "C:\opt\Anaconda3\Lib\site-packages\py4j\protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init.
: java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable
at com.amazonaws.services.glue.util.Job$.init(Job.scala:96)
at com.amazonaws.services.glue.util.Job.init(Job.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:75)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.ClassNotFoundException: com.typesafe.config.ConfigMergeable
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 14 more
23/11/14 22:12:22 INFO SparkContext: Invoking stop() from shutdown hook
23/11/14 22:12:22 INFO SparkContext: SparkContext is stopping with exitCode 0.
Process finished with exit code 1
字符串
1条答案
按热度按时间vsdwdz231#
我能够解决它。整个想法是,/opt/spark/jars和./aws-glue-libs/jarsv 1中的jar文件应该匹配。我复制了两个文件夹中的jar文件,使其相同,问题消失了