如何安装apache-spark 2.2.0与自制在Mac上

2fjabf4q  于 2023-01-26  发布在  Apache
关注(0)|答案(6)|浏览(153)

“$brew install apache-spark'让我得到版本2.3.x。'$brew search apache-spark'和'$brew info apache-spark'不提供一个选项来安装一个不同的版本。有可能得到一个不同的版本与homebrew?

jucafojl

jucafojl1#

运行这些命令(假设您已经通过Homebrew安装了apache-spark)

cd "$(brew --repo homebrew/core)"
git log Formula/apache-spark.rb

例如,2.2.0版本:
...
提交
作者:ilovezfs
日期:2017年7月11日星期二22:19:12 - 07:00
高速缓存-Spark2.2.0(编号15507)
....

git checkout -b  apache-spark-2.2.0 bdf68bd79ebd16a70b7a747e027afbe5831f9cc3
brew unlink apache-spark
HOMEBREW_NO_AUTO_UPDATE=1 brew install apache-spark

清理

git checkout master
git branch -d apache-spark-2.2.0

检查/切换:

brew list apache-spark --versions
brew switch apache-spark 2.2.0
toe95027

toe950272#

我有同样的问题,当我通过自制安装,默认情况下,它只能找到apacheSpark2.3.0公式,无法找到2.2.0,甚至删除回购。
因此,我已经备份了路径中现有的apache-spark. rb(版本2.3.0):/usr/local/homebrew/Library/水龙头/homebrew/homebrew-core/Formula,然后用以下内容覆盖:

class ApacheSpark < Formula
  desc "Engine for large-scale data processing"
  homepage "https://spark.apache.org/"
  url "https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz"
  version "2.2.0"
  sha256 "97fd2cc58e08975d9c4e4ffa8d7f8012c0ac2792bcd9945ce2a561cf937aebcc"
  head "https://github.com/apache/spark.git"

  bottle :unneeded

  def install
    # Rename beeline to distinguish it from hive's beeline
    mv "bin/beeline", "bin/spark-beeline"

    rm_f Dir["bin/*.cmd"]
    libexec.install Dir["*"]
    bin.write_exec_script Dir["#{libexec}/bin/*"]
  end

  test do
    assert_match "Long = 1000", pipe_output(bin/"spark-shell", "sc.parallelize(1 to 1000).count()")
  end
end

然后按照上述过程重新安装,我有2.2.0和2.3.0与开关设施。
希望有帮助。

anauzrmj

anauzrmj3#

我需要在我的MacBook上安装Apache Spark 2.4.0版本。但在Brew列表中不再提供,但您仍然可以进行。
通过brew install apache-spark安装最新的Spark。假设它安装了apache-spark-3.0.1
完成后,执行brew edit apache-spark并编辑Pachecos-spark. rb,如下所示

class ApacheSpark < Formula
  desc "Engine for large-scale data processing"
  homepage "https://spark.apache.org/"
  url "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
  mirror "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
  version "2.4.0"
  sha256 "c93c096c8d64062345b26b34c85127a6848cff95a4bb829333a06b83222a5cfa"
  license "Apache-2.0"
  head "https://github.com/apache/spark.git"

  bottle :unneeded

  depends_on "openjdk@8"

  def install
    # Rename beeline to distinguish it from hive's beeline
    mv "bin/beeline", "bin/spark-beeline"

    rm_f Dir["bin/*.cmd"]
    libexec.install Dir["*"]
    bin.install Dir[libexec/"bin/*"]
    bin.env_script_all_files(libexec/"bin", JAVA_HOME: Formula["openjdk@8"].opt_prefix)
  end

  test do
    assert_match "Long = 1000",
      pipe_output(bin/"spark-shell --conf spark.driver.bindAddress=127.0.0.1",
                  "sc.parallelize(1 to 1000).count()")
  end
end

现在使用brew uninstall apache-spark再次卸载spark使用brew install apache-spark再次安装

    • 结果**
% spark-shell
2021-02-09 19:27:11 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.17:4040
Spark context available as 'sc' (master = local[*], app id = local-1612927640472).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.
camsedfj

camsedfj4#

为了子孙后代:因为公式(https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz)中的url不再有效,所以恢复一个旧的brew提交是没有意义的。这也意味着2.2.1的brew公式也不能工作。
至少,您需要将url更新为http://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz(如@juanpaolo所述)。
今天要通过Homebrew安装Spark 2.2.0,
1.获取2.2.0公式(https://github.com/Homebrew/homebrew-core/blob/bdf68bd79ebd16a70b7a747e027afbe5831f9cc3/Formula/apache-spark.rb
1.将第4行中的url从https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz更新为http://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz

  1. brew install <path-to-updated-formula>
    • TLDR/适用于懒人:**
brew install https://gist.githubusercontent.com/eddies/bc148d83b1fc5555520d0cdf2dff8553/raw/c7ce091a083cacb3519502860695b56b0b806070/apache-spark.rb
    • 或者,通过冲泡龙头:**
brew tap eddies/spark-tap
brew install apache-spark@2.2.0
oxiaedzo

oxiaedzo5#

你甚至可以搜索apache-spark可用的公式列表:

brew search apache-spark

然后点击:

brew tap eddies/spark-tap

然后安装可用的特定版本:

brew install apache-spark@2.3.2
yptwkmov

yptwkmov6#

你可以简单地Unistall任何版本的scala,你已经在你的mac第一。
然后从你的终端是macbook类型brew install apache-spark@2.2.0,这将安装spark版本2.2.0在你的mac上。

相关问题