我正在用spark submit运行一个jar文件,但我一直遇到以下错误:
Error: Failed to load class antarctic.DataQuality.
这是命令:
spark-submit --class antarctic.DataQuality --master local[*] --deploy-mode client --jars "path/to/file.jar" "arg 1" "arg 2" "arg 3" "arg 4"
这是scala文件的结构:scala结构
命令正在target/scala-2.11中运行
我还定义了以下环境变量:
JAVA_HOME: C:\Program Files\Java\jdk1.8.0_251
HADOOP_HOME: C:\winutils
SPARK_HOME: C:\Users\Spark
SPARK_USER: C:\Users\Spark
内部版本.sbt:
name := "DataQuality"
version := "0.1"
scalaVersion := "2.11.8"
val sparkVersion = "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-mllib" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.spark" %% "spark-hive" % sparkVersion
)
// https://github.com/awslabs/deequ
libraryDependencies += "com.amazon.deequ" % "deequ" % "1.0.2-antarctic" from "file:///C:/Users/AKAINIX ANALYTICS/Documents/Lucas/Antarctic/Bitbucket/plataforma-dataquality/Backend/deequ/target/deequ-1.0.2-antarctic.jar"
//https://github.com/spray/spray-json
//libraryDependencies += "io.spray" %% "spray-json" % "1.3.5"
//https://github.com/scopt/scopt
libraryDependencies += "com.github.scopt" %% "scopt" % "4.0.0-RC2"
//https://circe.github.io/circe/
val circeVersion = "0.7.0"
libraryDependencies ++= Seq(
"io.circe" %% "circe-core" % circeVersion,
"io.circe" %% "circe-generic" % circeVersion,
"io.circe" %% "circe-parser" % circeVersion
)
// https://mvnrepository.com/artifact/mysql/mysql-connector-java
libraryDependencies += "mysql" % "mysql-connector-java" % "8.0.19"
// https://mvnrepository.com/artifact/org.postgresql/postgresql
libraryDependencies += "org.postgresql" % "postgresql" % "42.2.5"
//https://github.com/lightbend/config
libraryDependencies += "com.typesafe" % "config" % "1.4.0"
// https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector
libraryDependencies += "org.mongodb.spark" %% "mongo-spark-connector" % "2.3.0"
暂无答案!
目前还没有任何答案,快来回答吧!