我试图通过为我的项目运行sbt assembly来制作一个胖jar。我得到以下错误:
[error] (root/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-core/jars/hadoop-mapreduce-client-core-2.2.0.jar:org/apache/hadoop/filecache/DistributedCache.class
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-core/jars/hadoop-core-2.0.0-mr1-cdh4.7.1.jar:org/apache/hadoop/filecache/DistributedCache.class
hadoop mapreduce客户端核心的distributedcache现在不推荐使用。在my build.sbt中,我包括:
"org.apache.hadoop" % "hadoop-client" % "2.0.0-mr1-cdh4.7.1" excludeAll(
ExclusionRule(organization = "javax.servlet"))
依赖关系如下:
org.apache.hadoop:hadoop-client:2.2.0
org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0
org.apache.hadoop:hadoop-mapreduce-client-core:2.2.0
我该怎么处理?
提前谢谢!
1条答案
按热度按时间falq053o1#
如果您打算删除mapreduce客户端应用程序的依赖项jar,那么在您依赖hadoop的情况下,该应用程序正在加载-client:2.2.0,只需添加不及物:
这将只包括hadoop-client:2.2.0 jar 并排除其所有依赖项。