scala org.apache.http.client.utils.URIBuilder在Spark3.2上失败

aydmsdu9 于 2023-04-06 发布在 Scala

关注(0)|答案(1)|浏览(145)

我正在尝试从Spark 2.4升级到Spark 3.2版本。
import org.apache.http.client.utils.URIBuilder在Spark 3.1上运行正常，在Spark 3.2上运行失败，错误为object client is not a member of package org.apache.http。

scala

来源：https://stackoverflow.com/questions/75935866/org-apache-http-client-utils-uribuilder-failing-on-spark3-2

1条答案

按热度按时间

ulmd4ohb1#

类org.apache.http.client.utils.URIBuilder来自"org.apache.httpcomponents" % "httpclient"
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient
你应该加上

libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.5.14"

到build.sbt。然后import org.apache.http.client.utils.URIBuilder将编译。
(In第5版https://mvnrepository.com/artifact/org.apache.httpcomponents.client5/httpclient5这个类不存在。）
以下是原因的解释。
在Spark 2.4.8中，"org.apache.httpcomponents" % "httpclient"位于根pom.xml（<dependencyManagement>）中，例如在spark-corepom.xml中不会被覆盖

<!--  org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.6</commons.httpclient.version>
...

<dependencyManagement>
  <dependencies>
    ...
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>${commons.httpclient.version}</version>
    </dependency>

https://github.com/apache/spark/blob/v2.4.8/pom.xml#L499-L503
https://github.com/apache/spark/blob/v2.4.8/core/pom.xml
在Spark 3.2.0中，此依赖项也存在于根pom.xml（<dependencyManagement>）中，但在例如spark-corepom.xml（<dependencies>）中被覆盖，因此作用域为test
一个二个一个一个
https://github.com/apache/spark/blob/v3.2.0/pom.xml#L622-L626
https://github.com/apache/spark/blob/v3.2.0/core/pom.xml#L370-L375
因此，如果您不仅在test范围内需要此依赖项，那么您应该手动添加它。
Differences between dependencyManagement and dependencies in Maven

赞(0）回复(0）举报 2023-04-06

我来回答

scala org.apache.http.client.utils.URIBuilder在Spark3.2上失败

1条答案

相关问题

热门标签

最新问答