我在pycharm本地运行时遇到此错误,并尝试了所有选项:
Caused by: java.io.IOException: Cannot run program "/usr/local/Cellar/apache-spark/3.0.1/libexec/bin": error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:209)
at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:132)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:105)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:131)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
./bash_profile:export spark_home=/usr/local/opt/apache spark/libexec/export pythonpath=/usr/local/opt/apache spark/libexec/python/lib/py4j-0.10.9-src.zip:/usr/local/opt/apache spark/libexec/python/:/usr/local/lib/python3.9:$pythonp$export path=$spark_home/bin/:$spark_home/python/lib/py4j-0.10.9-src.zip:$path#export路径=$spark\u home/python:$path
ls-轻轨/usr/本地/opt/apache spark:
/usr/local/opt/apache-spark -> ../Cellar/apache-spark/3.0.1
pycharm中的python解释程序:/usr/local/bin/python3
在我的代码中:
if __name__ == '__main__':
#import os
#import sys
#os.environ['SPARK_HOME'] = "/usr/local/opt/apache-spark/libexec/"
#sys.path.append("/usr/local/opt/apache-spark/libexec/python")
#sys.path.append("/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip")
#findspark.init()
#conf = SparkConf()
#conf.set("fs.defaultFS", "file:///")
spark = SparkSession.builder.master("local").appName("SyslogMaskUtility").getOrCreate()
sc = spark.sparkContext
#sc.setLogLevel("WARN")
rdd_raw = sc.textFile('/Users/abcd/PycharmProjects/SyslogToJson/SyslogParser/syslog_event.txt')
print(rdd_raw.count())
spark.stop()
我跟着:https://medium.com/beeranddiapers/installing-apache-spark-on-mac-os-ce416007d79f
参考:spark安装看起来还可以,但是在运行程序时,我遇到了环境变量的问题。这是bash\u的资料吗?
/usr/local/opt/apache spark/libexec/下的所有目录和文件都具有以下所有权限:
drwxrwxrwx 13 abcd admin 416 Oct 29 17:34 bin
有什么需要帮忙的吗?因为我正在努力解决这个问题。当我从pyspark命令行运行时,同样的代码也能工作。
谢谢。
暂无答案!
目前还没有任何答案,快来回答吧!