flink连接器cassandra-硬错误

ubbxdtey  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(224)

我在本地环境中向flink（v1.0.3）提交新作业时遇到此错误。
原因：java.lang.noclassdeffounderror:org/apache/flink/streaming/runtime/operators/checkpointcommitter位于org.apache.flink.streaming.connectors.cassandra.cassandrasink.addsink（cassandrasink）。java:164)在com..yyy.sample.backend.flink.aaaaaa.main（aaaaa。java:99)位于sun.reflect.nativemethodaccessorimpl.invoke0（本机方法）sun.reflect.nativemethodaccessorimpl.invoke（nativemethodaccessorimpl。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke（delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke（方法。java:498)在org.apache.flink.client.program.packagedprogram.callmain方法（packagedprogram。java:505)在org.apache.flink.client.program.packagedprogram.invokeinteractivemodeforexecution（packagedprogram。java:403)位于org.apache.flink.client.program.optimizerplanenvironment.getoptimizedplan（optimizerplanenvironment）。java:80) ... 33更多关于github的源代码
https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators
我认为类应该是可用的，并且应该打包在flink-streaming-java.jar中，但是一旦导入了文件，类就不在了
缺少检查点提交者
有什么想法吗？
pom.xml文件
http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0

<groupId>com.AAAAA.BBB</groupId>
<artifactId>sample-backend-flink</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<name>Flink Quickstart Job</name>
<url>http://www.myorganization.org</url>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <flink.version>1.0.3</flink.version>
    <cassandra.version>2.2.7</cassandra.version>
    <driver.version>3.0.0</driver.version>
</properties>

<repositories>
    <repository>
        <id>apache.snapshots</id>
        <name>Apache Development Snapshot Repository</name>
        <url>https://repository.apache.org/content/repositories/snapshots/</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>${flink.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.10</artifactId>
        <version>${flink.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_2.10</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-core</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-kafka-0.8_2.10</artifactId>
        <version>1.0.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-cassandra_2.10</artifactId>
        <version>1.0.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-redis_2.10</artifactId>
        <version>1.1-SNAPSHOT</version>
    </dependency>
</dependencies>

<profiles>
    <profile>
        <!-- Profile for packaging correct JAR files -->
        <id>build-jar</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <dependencies>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-java</artifactId>
                <version>${flink.version}</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-java_2.10</artifactId>
                <version>${flink.version}</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-scala_2.10</artifactId>
                <version>${flink.version}</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-clients_2.10</artifactId>
                <version>${flink.version}</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-core</artifactId>
                <version>${flink.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-connector-kafka-0.8_2.10</artifactId>
                <version>1.0.2</version>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-connector-cassandra_2.10</artifactId>
                <version>1.1-SNAPSHOT</version>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-connector-redis_2.10</artifactId>
                <version>1.1-SNAPSHOT</version>
            </dependency>
        </dependencies>

        <build>
            <plugins>
                <!-- disable the exclusion rules -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>2.4.1</version>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                            <configuration>
                                <artifactSet>
                                    <excludes combine.self="override"></excludes>
                                </artifactSet>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.1</version>
            <executions>
                <!-- Run shade goal on package phase -->
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <artifactSet>
                            <excludes><exclude>org.apache.flink:flink-annotations</exclude>
                                <exclude>org.apache.flink:flink-shaded-hadoop1</exclude>
                                <exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
                                <exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
                                <exclude>org.apache.flink:flink-core</exclude>
                                <exclude>org.apache.flink:flink-java</exclude>
                                <exclude>org.apache.flink:flink-scala_2.10</exclude>
                                <exclude>org.apache.flink:flink-runtime_2.10</exclude>
                                <exclude>org.apache.flink:flink-optimizer_2.10</exclude>
                                <exclude>org.apache.flink:flink-clients_2.10</exclude>
                                <exclude>org.apache.flink:flink-avro_2.10</exclude>
                                <exclude>org.apache.flink:flink-examples-batch_2.10</exclude>
                                <exclude>org.apache.flink:flink-examples-streaming_2.10</exclude>
                                <exclude>org.apache.flink:flink-streaming-java_2.10</exclude>

                                <exclude>org.scala-lang:scala-library</exclude>
                                <exclude>org.scala-lang:scala-compiler</exclude>
                                <exclude>org.scala-lang:scala-reflect</exclude>
                                <exclude>com.amazonaws:aws-java-sdk</exclude>
                                <exclude>com.typesafe.akka:akka-actor_*</exclude>
                                <exclude>com.typesafe.akka:akka-remote_*</exclude>
                                <exclude>com.typesafe.akka:akka-slf4j_*</exclude>
                                <exclude>io.netty:netty-all</exclude>
                                <exclude>io.netty:netty</exclude>
                                <exclude>commons-fileupload:commons-fileupload</exclude>
                                <exclude>org.apache.avro:avro</exclude>
                                <exclude>commons-collections:commons-collections</exclude>
                                <exclude>org.codehaus.jackson:jackson-core-asl</exclude>
                                <exclude>org.codehaus.jackson:jackson-mapper-asl</exclude>
                                <exclude>com.thoughtworks.paranamer:paranamer</exclude>
                                <exclude>org.xerial.snappy:snappy-java</exclude>
                                <exclude>org.apache.commons:commons-compress</exclude>
                                <exclude>org.tukaani:xz</exclude>
                                <exclude>com.esotericsoftware.kryo:kryo</exclude>
                                <exclude>com.esotericsoftware.minlog:minlog</exclude>
                                <exclude>org.objenesis:objenesis</exclude>
                                <exclude>com.twitter:chill_*</exclude>
                                <exclude>com.twitter:chill-java</exclude>
                                <exclude>com.twitter:chill-avro_*</exclude>
                                <exclude>com.twitter:chill-bijection_*</exclude>
                                <exclude>com.twitter:bijection-core_*</exclude>
                                <exclude>com.twitter:bijection-avro_*</exclude>
                                <exclude>commons-lang:commons-lang</exclude>
                                <exclude>junit:junit</exclude>
                                <exclude>de.javakaffee:kryo-serializers</exclude>
                                <exclude>joda-time:joda-time</exclude>
                                <exclude>org.apache.commons:commons-lang3</exclude>
                                <exclude>org.slf4j:slf4j-api</exclude>
                                <exclude>org.slf4j:slf4j-log4j12</exclude>
                                <exclude>log4j:log4j</exclude>
                                <exclude>org.apache.commons:commons-math</exclude>
                                <exclude>org.apache.sling:org.apache.sling.commons.json</exclude>
                                <exclude>commons-logging:commons-logging</exclude>
                                <exclude>commons-codec:commons-codec</exclude>
                                <exclude>com.fasterxml.jackson.core:jackson-core</exclude>
                                <exclude>com.fasterxml.jackson.core:jackson-databind</exclude>
                                <exclude>com.fasterxml.jackson.core:jackson-annotations</exclude>
                                <exclude>stax:stax-api</exclude>
                                <exclude>com.typesafe:config</exclude>
                                <exclude>org.uncommons.maths:uncommons-maths</exclude>
                                <exclude>com.github.scopt:scopt_*</exclude>
                                <exclude>commons-io:commons-io</exclude>
                                <exclude>commons-cli:commons-cli</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>org.apache.flink:*</artifact>
                                <excludes>
                                    <!-- exclude shaded google but include shaded curator -->
                                    <exclude>org/apache/flink/shaded/com/**</exclude>
                                    <exclude>web-docs/**</exclude>
                                </excludes>
                            </filter>
                            <filter>
                                <!-- Do not copy the signatures in the META-INF folder.
                                Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <!-- add Main-Class to manifest file -->
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.demandware.unified.sample.backend.flink.PhysicalInventory</mainClass>
                            </transformer>
                        </transformers>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
                <source>1.8</source> <!-- If you want to use Java 8, change this to "1.8" -->
                <target>1.8</target> <!-- If you want to use Java 8, change this to "1.8" -->
            </configuration>
        </plugin>
    </plugins>
</build>
cassandraapache-flink

1条答案

按热度按时间
yebdmbv4

yebdmbv41#

我也有类似的问题。我所修复的是确保源代码中使用的kafkaconsumer/producer与您所依赖的匹配。
flinkkafkaconsumer的不同版本之间似乎存在冲突。
使用此导入时

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka-0.8_2.10</artifactId>
    <version>1.0.2</version>
</dependency>

确保您使用的是flinkkafkaconsumer08，而不是flinkkafkaconsumer（如果您使用的是producer，则使用producer）
对我来说奇怪的是，我的ide不能解析flinkkafkaconsumer08，但它使用构建工具编译并正确运行。

赞(0）分享  回复(0）举报  2021-06-25
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com