class ApacheSpark < Formula
desc "Engine for large-scale data processing"
homepage "https://spark.apache.org/"
url "https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz"
version "2.2.0"
sha256 "97fd2cc58e08975d9c4e4ffa8d7f8012c0ac2792bcd9945ce2a561cf937aebcc"
head "https://github.com/apache/spark.git"
bottle :unneeded
def install
# Rename beeline to distinguish it from hive's beeline
mv "bin/beeline", "bin/spark-beeline"
rm_f Dir["bin/*.cmd"]
libexec.install Dir["*"]
bin.write_exec_script Dir["#{libexec}/bin/*"]
end
test do
assert_match "Long = 1000", pipe_output(bin/"spark-shell", "sc.parallelize(1 to 1000).count()")
end
end
class ApacheSpark < Formula
desc "Engine for large-scale data processing"
homepage "https://spark.apache.org/"
url "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
mirror "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
version "2.4.0"
sha256 "c93c096c8d64062345b26b34c85127a6848cff95a4bb829333a06b83222a5cfa"
license "Apache-2.0"
head "https://github.com/apache/spark.git"
bottle :unneeded
depends_on "openjdk@8"
def install
# Rename beeline to distinguish it from hive's beeline
mv "bin/beeline", "bin/spark-beeline"
rm_f Dir["bin/*.cmd"]
libexec.install Dir["*"]
bin.install Dir[libexec/"bin/*"]
bin.env_script_all_files(libexec/"bin", JAVA_HOME: Formula["openjdk@8"].opt_prefix)
end
test do
assert_match "Long = 1000",
pipe_output(bin/"spark-shell --conf spark.driver.bindAddress=127.0.0.1",
"sc.parallelize(1 to 1000).count()")
end
end
% spark-shell
2021-02-09 19:27:11 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.17:4040
Spark context available as 'sc' (master = local[*], app id = local-1612927640472).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.
6条答案
按热度按时间jucafojl1#
运行这些命令(假设您已经通过Homebrew安装了apache-spark)
例如,2.2.0版本:
...
提交
作者:ilovezfs
日期:2017年7月11日星期二22:19:12 - 07:00
高速缓存-Spark2.2.0(编号15507)
....
清理
检查/切换:
toe950272#
我有同样的问题,当我通过自制安装,默认情况下,它只能找到apacheSpark2.3.0公式,无法找到2.2.0,甚至删除回购。
因此,我已经备份了路径中现有的apache-spark. rb(版本2.3.0):/usr/local/homebrew/Library/水龙头/homebrew/homebrew-core/Formula,然后用以下内容覆盖:
然后按照上述过程重新安装,我有2.2.0和2.3.0与开关设施。
希望有帮助。
anauzrmj3#
我需要在我的MacBook上安装Apache Spark 2.4.0版本。但在Brew列表中不再提供,但您仍然可以进行。
通过
brew install apache-spark
安装最新的Spark。假设它安装了apache-spark-3.0.1完成后,执行
brew edit apache-spark
并编辑Pachecos-spark. rb,如下所示现在使用
brew uninstall apache-spark
再次卸载spark使用brew install apache-spark
再次安装camsedfj4#
为了子孙后代:因为公式(https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz)中的url不再有效,所以恢复一个旧的brew提交是没有意义的。这也意味着2.2.1的brew公式也不能工作。
至少,您需要将url更新为http://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz(如@juanpaolo所述)。
今天要通过Homebrew安装Spark 2.2.0,
1.获取2.2.0公式(https://github.com/Homebrew/homebrew-core/blob/bdf68bd79ebd16a70b7a747e027afbe5831f9cc3/Formula/apache-spark.rb)
1.将第4行中的url从https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz更新为http://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz
brew install <path-to-updated-formula>
oxiaedzo5#
你甚至可以搜索apache-spark可用的公式列表:
然后点击:
然后安装可用的特定版本:
yptwkmov6#
你可以简单地Unistall任何版本的scala,你已经在你的mac第一。
然后从你的终端是macbook类型
brew install apache-spark@2.2.0
,这将安装spark版本2.2.0在你的mac上。