我是新来的。我正在尝试从中创建一个spark会话 pyspark.sql
以加载.csv文件。但是,每次我试图执行第2行(如下所示),命令都会持续执行几个小时&似乎永远不会生成代码的其他行。代码如下:
from pyspark.sql import SparkSession
sp = SparkSession.builder.appName("solution").config("spark.some.config.option", "some-value").getOrCreate()
df = sp.read.csv('walmart_stock.csv', header= True, inferSchema= True)
df.columns
另外,如果我在等待很长时间后终止内核,则会出现以下异常:
<ipython-input-23-16c3797ce83f> in <module>
----> 1 sp = SparkSession.builder.appName("solution").config("spark.some.config.option", "some-value").getOrCreate()
~\anaconda3\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
184 sparkConf.set(key, value)
185 # This SparkContext may be an existing one.
--> 186 sc = SparkContext.getOrCreate(sparkConf)
187 # Do not update `SparkConf` for existing `SparkContext`, as it's shared
188 # by all sessions.
~\anaconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
369 with SparkContext._lock:
370 if SparkContext._active_spark_context is None:
--> 371 SparkContext(conf=conf or SparkConf())
372 return SparkContext._active_spark_context
373
~\anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
126 " is not allowed as it is a security risk.")
127
--> 128 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
129 try:
130 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
~\anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
318 with SparkContext._lock:
319 if not SparkContext._gateway:
--> 320 SparkContext._gateway = gateway or launch_gateway(conf)
321 SparkContext._jvm = SparkContext._gateway.jvm
322
~\anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
100 # Wait for the file to appear, or for the process to exit, whichever happens first.
101 while not proc.poll() and not os.path.isfile(conn_info_file):
--> 102 time.sleep(0.1)
103
104 if not os.path.isfile(conn_info_file):
你能告诉我有什么问题吗?
暂无答案!
目前还没有任何答案,快来回答吧!