pyspark在mac上本地运行:原因:java.io.ioexception:cannot run program“/usr/local/cellar/apache spark/3.0.1/libexec/bin”

t3irkdon  于 2021-07-14  发布在  Spark
关注(0)|答案(0)|浏览(245)

我在pycharm本地运行时遇到此错误,并尝试了所有选项:

Caused by: java.io.IOException: Cannot run program "/usr/local/Cellar/apache-spark/3.0.1/libexec/bin": error=13, Permission denied
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:209)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:132)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:105)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
    at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:131)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

./bash_profile:export spark_home=/usr/local/opt/apache spark/libexec/export pythonpath=/usr/local/opt/apache spark/libexec/python/lib/py4j-0.10.9-src.zip:/usr/local/opt/apache spark/libexec/python/:/usr/local/lib/python3.9:$pythonp$export path=$spark_home/bin/:$spark_home/python/lib/py4j-0.10.9-src.zip:$path#export路径=$spark\u home/python:$path
ls-轻轨/usr/本地/opt/apache spark:

/usr/local/opt/apache-spark -> ../Cellar/apache-spark/3.0.1

pycharm中的python解释程序:/usr/local/bin/python3
在我的代码中:

if __name__ == '__main__':
    #import os
    #import sys
    #os.environ['SPARK_HOME'] = "/usr/local/opt/apache-spark/libexec/"
    #sys.path.append("/usr/local/opt/apache-spark/libexec/python")
    #sys.path.append("/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip")
    #findspark.init()
    #conf = SparkConf()
    #conf.set("fs.defaultFS", "file:///")
    spark = SparkSession.builder.master("local").appName("SyslogMaskUtility").getOrCreate()
    sc = spark.sparkContext
    #sc.setLogLevel("WARN")
    rdd_raw = sc.textFile('/Users/abcd/PycharmProjects/SyslogToJson/SyslogParser/syslog_event.txt')
    print(rdd_raw.count())
    spark.stop()

我跟着:https://medium.com/beeranddiapers/installing-apache-spark-on-mac-os-ce416007d79f
参考:spark安装看起来还可以,但是在运行程序时,我遇到了环境变量的问题。这是bash\u的资料吗?
/usr/local/opt/apache spark/libexec/下的所有目录和文件都具有以下所有权限:

drwxrwxrwx   13 abcd  admin   416 Oct 29 17:34 bin

有什么需要帮忙的吗?因为我正在努力解决这个问题。当我从pyspark命令行运行时,同样的代码也能工作。
谢谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题