scala依赖问题与烫伤和hadoop与sbt程序集

pgx2nnw8 于 2021-06-03 发布在 Hadoop

关注(0)|答案(1)|浏览(438)

我想和你建立一个远大的关系 sbt 一个简单的hadoop作业，我试图在amazonemr上运行它。但是当我跑的时候 sbt assembly 我得到以下错误：

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/trenthauck/.ivy2/cache/org.mortbay.jetty/jsp-2.1/jars/jsp-2.1-6.1.14.jar:org/apache/jasper/compiler/Node$ChildInfo.class
[error] /Users/trenthauck/.ivy2/cache/tomcat/jasper-compiler/jars/jasper-compiler-5.5.12.jar:org/apache/jasper/compiler/Node$ChildInfo.class
[error] Total time: 10 s, completed Sep 14, 2013 4:49:24 PM

我在这里试图遵循这个建议https://groups.google.com/forum/#!topic/simple build tool/tzkq5tioiqm，但是它不起作用。
我的build.sbt看起来像：

import AssemblyKeys._

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", "jasper", xs @ _*) => MergeStrategy.last
    case x => old(x)
  }
}

assemblySettings

name := "Scaling Play"

version := "SNAPSHOT-0.1"

scalaVersion := "2.10.1"

libraryDependencies ++= Seq(
  "com.twitter" % "scalding-core_2.10" % "0.8.8",
  "com.twitter" % "scalding-args_2.10" % "0.8.8",
  "com.twitter" % "scalding-date_2.10" % "0.8.8",
  "org.apache.hadoop" % "hadoop-core" % "1.0.0"
  )

hadoop scala sbt scalding sbt-assembly

来源：https://stackoverflow.com/questions/18806807/dependency-issue-with-scalding-and-hadoop-with-sbt-assembly

1条答案

按热度按时间

cxfofazt1#

指令的顺序很重要。您可以更新部件设置，以便在一行之后再次覆盖它。首先定义assemblysettings，然后更新它将解决这个问题。
更新的build.sbt：

import AssemblyKeys._

assemblySettings

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", "jasper", xs @ _*) => MergeStrategy.last
    case x => old(x)
  }
}

…

在此之后，您将发现有更多的冲突类和其他文件。在这种情况下，您将需要以下合并：

case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
case PathList("project.clj") => MergeStrategy.last
case PathList("overview.html") => MergeStrategy.last
case x => old(x)

请注意，对类文件使用合并策略可能会产生问题，这是由特定类的不兼容版本引起的。如果是这样的话，那么你的问题就更大了，因为依赖项之间是不兼容的。然后，您必须删除依赖项并找到/生成兼容的版本。

赞(0）回复(0）举报 2021-06-03

我来回答

scala依赖问题与烫伤和hadoop与sbt程序集

1条答案

相关问题

热门标签

最新问答