我正在尝试使用spark连接到hbase,我想使用hortonworks shc,但是我不能为shc core导入sbt依赖项(尽管我可以导入shc)。下面是build.sbt快照-
name := "untitled"
version := "0.1"
scalaVersion := "2.11.12"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.2"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.2"
resolvers += "Hortonworks Repository" at "http://repo.hortonworks.com/content/repositories/releases/"
// https://mvnrepository.com/artifact/com.hortonworks.shc/shc
libraryDependencies += "com.hortonworks.shc" % "shc" % "1.1.0.3.0.2.0-50" pomOnly()
// https://mvnrepository.com/artifact/com.hortonworks.shc/shc-core
libraryDependencies += "com.hortonworks.shc" % "shc-core" % "1.1.0.3.0.2.0-50"
我得到以下错误(开始几行不包括在内)-
[info] Resolving org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook ...
[info] Resolving org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook ...
[warn] module not found: org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook
[warn] ==== local: tried
[warn] /home/ashish/.ivy2/local/org.apache.hadoop/hadoop-aws/working@ashish-HP-Notebook/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/ashish/.sbt/preloaded/org.apache.hadoop/hadoop-aws/working@ashish-HP-Notebook/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/ashish/.sbt/preloaded/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[warn] ==== Hortonworks Repository: tried
[warn] http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[info] Resolving jline#jline;2.14.3 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.github.stephenc.jcip#jcip-annotations;1.0-1: com.github.stephenc.jcip#jcip-annotations;1.0-1!jcip-annotations.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/github/stephenc/jcip/jcip-annotations/1.0-1/jcip-annotations-1.0-1.pom
[warn] :: com.google.guava#guava;13.0.1: com.google.guava#guava;13.0.1!guava.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.pom
[warn] :: com.google.inject.extensions#guice-assistedinject;3.0: com.google.inject.extensions#guice-assistedinject;3.0!guice-assistedinject.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.pom
[warn] :: ch.qos.logback#logback-classic;1.0.9: ch.qos.logback#logback-classic;1.0.9!logback-classic.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/ch/qos/logback/logback-classic/1.0.9/logback-classic-1.0.9.pom
[warn] :: ch.qos.logback#logback-core;1.0.9: ch.qos.logback#logback-core;1.0.9!logback-core.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/ch/qos/logback/logback-core/1.0.9/logback-core-1.0.9.pom
[warn] :: org.apache.thrift#libthrift;0.9.0: org.apache.thrift#libthrift;0.9.0!libthrift.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/apache/thrift/libthrift/0.9.0/libthrift-0.9.0.pom
[warn] :: it.unimi.dsi#fastutil;6.5.6: it.unimi.dsi#fastutil;6.5.6!fastutil.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/it/unimi/dsi/fastutil/6.5.6/fastutil-6.5.6.pom
[warn] :: org.antlr#antlr-runtime;3.5.2: org.antlr#antlr-runtime;3.5.2!antlr-runtime.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.pom
[warn] :: sqlline#sqlline;1.2.0: sqlline#sqlline;1.2.0!sqlline.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/sqlline/sqlline/1.2.0/sqlline-1.2.0.pom
[warn] :: org.iq80.snappy#snappy;0.3: org.iq80.snappy#snappy;0.3!snappy.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/iq80/snappy/snappy/0.3/snappy-0.3.pom
[warn] :: org.apache.commons#commons-csv;1.0: org.apache.commons#commons-csv;1.0!commons-csv.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/apache/commons/commons-csv/1.0/commons-csv-1.0.pom
[warn] :: org.ow2.asm#asm;5.0.4: org.ow2#ow2;1.3!ow2.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/ow2/ow2/1.3/ow2-1.3.pom
[warn] :: org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.apache.hadoop:hadoop-aws:working@ashish-HP-Notebook
[warn] +- org.apache.hbase:hbase-client:2.0.0.3.0.2.0-50
[warn] +- org.apache.hbase:hbase-replication:2.0.0.3.0.2.0-50
[warn] +- org.apache.hbase:hbase-mapreduce:2.0.0.3.0.2.0-50
[warn] +- org.apache.phoenix:phoenix-core:5.0.0.3.0.2.0-50
[warn] +- com.hortonworks.shc:shc-core:1.1.0.3.0.2.0-50 (/home/ashish/sparkprojects/untitled/build.sbt#L20-21)
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.github.stephenc.jcip#jcip-annotations;1.0-1: com.github.stephenc.jcip#jcip-annotations;1.0-1!jcip-annotations.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/github/stephenc/jcip/jcip-annotations/1.0-1/jcip-annotations-1.0-1.pom
[error] unresolved dependency: com.google.guava#guava;13.0.1: com.google.guava#guava;13.0.1!guava.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.pom
[error] unresolved dependency: com.google.inject.extensions#guice-assistedinject;3.0: com.google.inject.extensions#guice-assistedinject;3.0!guice-assistedinject.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.pom
我使用的是spark2.2.2和sbt.13以及ubuntu。
而且,当我在windows中执行此操作时,没有任何错误。
2条答案
按热度按时间plupiseo1#
看起来它是一个scala依赖项,所以应该让sbt像spark一样处理scala版本。使用双%%
eqoofvh92#
似乎库导入的指定不正确。根据文件,只是进口
shc-core
应满足以下要求:另外,请注意
pomOnly()
sbt将使处理程序跳过此依赖项的任何jar lib或工件。添加它将只从聚合pom获取元数据。希望这有帮助。