java.lang.noclassdeffounderror提交flink作业时出错

xjreopfe  于 2021-06-21  发布在  Flink
关注(0)|答案(2)|浏览(576)

我正在尝试提交一个使用scala 2.11创建的flink作业,该作业在本地flink集群中使用twitter流api,方法是在命令行中运行: flink run -c org.myClass C:\path\to\jarFile.jar 并得到以下错误:

2019-06-09 23:40:47,758 WARN  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler    - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2019-06-09 23:40:47,762 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler    - Unhandled exception.
org.apache.flink.client.program.ProgramInvocationException: The program caused an error: 
    at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
    at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:80)
    at org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:126)
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$6(JarRunHandler.java:142)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/twitter/TwitterSource$EndpointInitializer
    at msciss.TwitterHashtagCounter.main(TwitterHashtagCounter.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
    at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
    ... 7 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.twitter.TwitterSource$EndpointInitializer
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 15 more

但是在程序中,我在build.sbt中设置了twittersource库:

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-connector-twitter" % flinkVersion,
  "commons-logging" % "commons-logging" % "1.2",
  "org.apache.logging.log4j" % "log4j-core" % "2.11.2",
  "org.apache.commons" % "commons-text" % "1.6")

该应用程序在intellij中也运行正常,sbt buld/package不会产生任何问题。如何解决此问题?

7gs2gvoe

7gs2gvoe1#

我有一个胖jar。当我分解它时,我可以看到连接器的依赖关系也是。然而,当我将jar作为flink作业提交时,我得到了一个classnotfoundexception。
原因是什么?

qlckcl4x

qlckcl4x2#

你需要使用 sbt assembly 插件或任何其他允许创建胖jar(uberjar)的插件。当前,您的包不包含外部库,并且flink连接器被视为外部库,因为它们不包括在标准二进制生成中。因此,实际上您正在创建的包不包含 twitter-connector ,但flink本身也没有,这就是为什么你得到 ClassNotFoundException .

相关问题