示例pyspark程序返回[winerror 2]系统找不到该文件

qeeaahzv  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(484)

下面是我要运行的代码。我已经为spark、hadoop、java和python设置了路径。使用Java8、Spark2.2.1和Hadoop2.7.5。

import random
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('MyFirstStandaloneApp')
sc = SparkContext(conf=conf)
NUM_SAMPLES = 20
def inside(p):
   x, y = random.random(), random.random()
   return x*x + y*y < 1

count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
         .filter(inside).count()
print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))

我收到的错误如下:

Traceback (most recent call last):
File "sample1.py", line 4, in <module>
 sc = SparkContext(conf=conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py", line 
    115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py", line 
         283, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\java_gateway.py", 
         line 80, in launch_gateway
        proc = Popen(command, stdin=PIPE, env=env)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 709, in __init__
      restore_signals, start_new_session)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 997, 
     in_execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
lrpiutwd

lrpiutwd1#

打开anaconda prompt-type=python-m ipykernel install--user
打开jupyter笔记本

jvidinwx

jvidinwx2#

解决方案我安装了两次spark,一个是apache的独立版本,另一个是anaconda的版本,导致路径出现问题。

相关问题