未能加载:com/amazon/deequ/checks/check

tzxcd3kk  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(413)

我正在构建一个spark应用程序来加载两个json文件,比较它们,并打印差异。我还尝试使用amazon库验证这些文件 aws deequ ,但我得到以下例外:

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/08/07 11:56:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: Failed to load com.deeq.CompareDataFrames: com/amazon/deequ/checks/Check
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please

当我把工作交给spark时:

./spark-submit --class com.deeq.CompareDataFrames--master 
spark://saif-VirtualBox:7077 ~/Downloads/deeq-trial-1.0-SNAPSHOT.jar

我使用ubuntu来托管spark,在我添加deequ来运行一些验证之前,它没有任何问题。我想知道在部署过程中是否遗漏了什么。这似乎不是一个众所周知的错误在互联网上。
代码:

import com.amazon.deequ.VerificationResult;
import com.amazon.deequ.VerificationSuite;
import com.amazon.deequ.checks.Check;
import com.amazon.deequ.checks.CheckLevel;
import com.amazon.deequ.checks.CheckStatus;
import com.amazon.deequ.constraints.Constraint;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import scala.Option;
import scala.Tuple2;
import scala.collection.mutable.ArraySeq;
import scala.collection.mutable.Seq;

public class CompareDataFrames {

    public static void main(String[] args) {
        SparkSession session = SparkSession.builder().appName("CompareDataFrames").getOrCreate();
        session.sparkContext().setLogLevel("ALL");

        StructType schema = DataTypes.createStructType(new StructField[]{
                DataTypes.createStructField("CUST_ID", DataTypes.StringType, true),
                DataTypes.createStructField("RECORD_LOCATOR_ID", DataTypes.StringType, true),
                DataTypes.createStructField("EVNT_ID", DataTypes.StringType, true)
        });

        Dataset<Row> first = session.read().option("multiline", "true").schema(schema).json("/home/saif/Downloads/FILE_DEV1.json");
        System.out.println("======= DataSet 1 =======");
        first.printSchema();
        first.show(false);

        Dataset<Row> second = session.read().option("multiline", "true").schema(schema).json("/home/saif/Downloads/FILE_DEV2.json");
        System.out.println("======= DataSet 2 =======");
        second.printSchema();
        second.show(false);

        // This will show all the rows which are present in the first dataset
        // but not present in the second dataset. But the comparison is at row
        // level and not at column level.
        System.out.println("======= Expect =======");
        first.except(second).show();
        StructType one = first.schema();
        JavaPairRDD<String, Row> pair1 = first.toJavaRDD().mapToPair((PairFunction<Row, String, Row>)
                row -> new Tuple2<>(row.getString(1), row));
        JavaPairRDD<String, Row> pair2 = second.toJavaRDD().mapToPair((PairFunction<Row, String, Row>)
                row -> new Tuple2<>(row.getString(1), row));
        System.out.println("======= Pair1 & Pair2 were created =======");
        JavaPairRDD<String, Row> subs = pair1.subtractByKey(pair2);
        JavaRDD<Row> rdd = subs.values();
        Dataset<Row> diff = session.createDataFrame(rdd, one);
        System.out.println("======= Diff Show =======");
        diff.show();

        Seq<Constraint> cons = new ArraySeq<>(0);
        VerificationResult vr = new VerificationSuite().onData(first)
                .addCheck(new Check(CheckLevel.Error(), "unit test", cons)
                        .isComplete("EVNT_ID", Option.empty())
                )
                .run();
        Seq<Check> checkSeq = new ArraySeq<>(0);
        if (vr.status() != CheckStatus.Success()) {
            Dataset<Row> vrr = vr.checkResultsAsDataFrame(session, vr, checkSeq);
            vrr.show(false);
        }

    }

}

Maven:

<dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.12</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-catalyst_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazon.deequ</groupId>
            <artifactId>deequ</artifactId>
            <version>1.0.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.13.3</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang.modules</groupId>
            <artifactId>scala-java8-compat_2.13</artifactId>
            <version>0.9.1</version>
        </dependency>
h6my8fg2

h6my8fg21#

请按照以下方法解决您的问题。
方法1。
spark提交 --jars 选项,将jar从下面的maven repo下载到您的机器上,https://mvnrepository.com/artifact/com.amazon.deequ/deequ/1.0.4 至 ~/Downloads/deequ-1.0.4.jar ```
./spark-submit --class com.deeq.CompareDataFrames --master
spark://saif-VirtualBox:7077 --jars ~/Downloads/deequ-1.0.4.jar ~/Downloads/deeq-trial-1.0-SNAPSHOT.jar

方法2。
spark提交 `--packages` 选项,

./spark-submit --class com.deeq.CompareDataFrames --master
spark://saif-VirtualBox:7077 --packages com.amazon.deequ:deequ:1.0.4 ~/Downloads/deeq-trial-1.0-SNAPSHOT.jar

笔记:
这个 `--repositories` 只有在默认情况下必须引用某个自定义存储库时,选项才是必需的如果 `--repositories` 在以下情况下不提供选项: `--packages` 选项时,提交操作将尝试在 `~/.ivy2/cache` ,  `~/.ivy2/jars` ,  `~/.m2/repository` 目录。如果找不到,则使用ivy从maven central下载并存储在 `~/.ivy2` 目录。
编辑1:
方法3:
如果上述解决方案1和2不起作用,则使用 `maven-shade-plugin` 建造 `uber jar` 然后继续 `spark-submit` . 使用下面的 `pom.xml` 用于使用 `maven-shade-plugin` . 添加下面的pom并重建jar并使用 `spark-submit` .

spark-submit --class com.deeq.CompareDataFrames --master
spark://saif-VirtualBox:7077 ~/Downloads/deeq-trial-1.0-SNAPSHOT.jar


4.0.0
com.deeq
deeq-trial-1.0-SNAPSHOT
1.0
Spark-3.0 Spark Application
https://maven.apache.org


codelds
https://code.lds.org/nexus/content/groups/main-repo


central
Maven Repository Switchboard
default
https://repo1.maven.org/maven2

false




<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
UTF-8
<scala.version>2.12.8</scala.version>
<java.version>1.8</java.version>
512m
<es.version>2.4.6</es.version>



org.apache.spark
spark-core_2.12
3.0.0


org.apache.spark
spark-streaming_2.12
3.0.0
provided


org.apache.spark
spark-sql_2.12
3.0.0


org.apache.spark
spark-catalyst_2.12
3.0.0


com.amazon.deequ
deequ
1.0.4


org.apache.logging.log4j
log4j-core
2.13.3


org.scala-lang.modules
scala-java8-compat_2.13
0.9.1





src/main/resources




net.alchim31.maven
scala-maven-plugin
3.2.2


eclipse-add-source

add-source



scala-compile-first
process-resources

compile



scala-test-compile-first
process-test-resources

testCompile



attach-scaladocs
verify

doc-jar




${scala.version}
incremental
true

-unchecked
-deprecation
-feature


-Xms1024m
-Xmx1024m
-XX:ReservedCodeCacheSize=${CodeCacheSize}


-source
${java.version}
-target
${java.version}
-Xlint:all,-serial,-path




org.apache.maven.plugins
maven-shade-plugin


package

shade




org.xerial.snappy
org.scala-lang.modules
org.scala-lang




:

META-INF/.SF
META-INF/
.DSA
META-INF/*.RSA





com.google.common
shaded.com.google.common








相关问题