flink中的java.lang.nosuchmethoderror

eoigrqb6  于 2021-06-25  发布在  Flink
关注(0)|答案(7)|浏览(702)

我尝试使用以下方法读取文件:

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");

我得到以下错误:

java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found

我使用的是flink版本1.3.2,java版本“1.8.0\u91”

q0qdq0h2

q0qdq0h21#

您是否在intellij或dashboard中遇到此错误,如果在intellij中遇到此错误,请确保在pom.xml中使用相同的flink版本,并在生成中添加如下依赖项着色

<build>

        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>

                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>

                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.codehaus.plexus.util</pattern>
                                    <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                    <excludes>
                                        <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                        <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                    </excludes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

        </plugins>

</build>>

确保在进行更改后在终端中运行maven clean install。另一方面,如果您只在dashboard而不是intellij中遇到此问题,请查看此处

qeeaahzv

qeeaahzv2#

你需要检查你的构建路径,确保lib在那里并且正确导入它们

ie3xauqp

ie3xauqp3#

出现错误“java.lang.nosuchmethoderror”的一个可能原因是,当我们使用不同版本的flink时,我们的系统上安装的是什么。对我来说,我有Flink1.3.2,我使用的版本是1.1.1。所以我更新了pom文件,使其具有相同的版本。

tzcvj98z

tzcvj98z4#

我面临着类似的问题,对我来说问题是flink次要版本不匹配。我的本地flink集群运行的是flink-1.8.0,我的代码预期的版本是flink-1.8.3。切换到新版本解决了这个问题。

ehxuflar

ehxuflar5#

与依赖项存在冲突。apacheflink默认情况下会将许多类加载到其类路径中。
请看这篇文章https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html 最后一节
使用maven shade插件解决与flink的依赖冲突
apacheflink默认情况下会将许多类加载到其类路径中。如果用户使用flink正在使用的库的不同版本,通常 

IllegalAccessExceptions

 或 

NoSuchMethodError

 是结果。
因此,我建议使用pom.xml,使用maven shade插件并添加正确的重新定位,如示例中所示

<relocation>
  <pattern>org.codehaus.plexus.util</pattern>
  <shadedPattern>org.shaded.plexus.util</shadedPattern>
  <excludes>
    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
  </excludes>
</relocation>
7z5jn7bk

7z5jn7bk6#

此错误的另一个来源可能是flink和应用程序代码(或使用scala的应用程序依赖项)之间的scala版本不匹配。
例如,在我的例子中,我使用的是flink1.7.1,我必须将scala依赖项从2.11更新到2.12;我更新了相关依赖项的articatid,如下所示:from flink-scala_2.11flink-scala_2.12 , flink-table_2.11flink-table_2.12 等等。
更多信息请参见此处。

7eumitmz

7eumitmz7#

对于那些在使用 Flink 1.3.2 ,以下是我成功构建的整个pom:

<build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.4.1</version>
            <configuration>
               <!-- get all project dependencies -->
               <descriptorRefs>
                  <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
               <!-- MainClass in mainfest make a executable jar -->
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
            <executions>
               <execution>
                  <id>make-assembly</id>
                  <!-- bind to the packaging phase -->
                  <phase>package</phase>
                  <goals>
                     <goal>single</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
         </plugin>
         <!--added newly-->
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
               <source>1.8</source>
               <target>1.8</target>
               <compilerId>jdt</compilerId>
            </configuration>
            <dependencies>
               <dependency>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-compiler-jdt</artifactId>
                  <version>0.21.0</version>
               </dependency>
            </dependencies>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                           <mainClass>FlinktoLambda</mainClass>
                        </transformer>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                           <resource>reference.conf</resource>
                        </transformer>
                     </transformers>
                     <relocations>
                        <relocation>
                           <pattern>org.codehaus.plexus.util</pattern>
                           <shadedPattern>org.shaded.plexus.util</shadedPattern>
                           <excludes>
                              <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                              <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                           </excludes>
                        </relocation>
                     </relocations>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   <name>my-app</name>
   <url>http://maven.apache.org</url>
   <dependencyManagement>
      <dependencies>
         <dependency>
            <groupId>FlinktoLambda</groupId>
            <artifactId>my-app</artifactId>
            <exclusions>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-remote_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-actor_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-slf4j_2.10</artifactId>
               </exclusion>
            </exclusions>
         </dependency>
      </dependencies>
   </dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-table_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-java</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-java_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-clients_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
   </dependencies>

请在课堂上相应地改变你的主课 shade 插件。

相关问题