这个问题在这里已经有了答案:
使用spark和hivesql执行多个查询的问题(1个答案)
11个月前关门了。
我正在使用pyspark==2.4.3,我只想运行一个hql文件
use myDatabaseName;
show tables;
我试过了
from os.path import expanduser, join, abspath
from pyspark.sql import SparkSession
from pyspark.sql import Row
# warehouse_location points to the default location for managed databases and tables
warehouse_location = abspath('spark-warehouse')
spark = SparkSession \
.builder \
.appName("Python Spark SQL Hive integration example") \
.config("spark.sql.warehouse.dir", warehouse_location) \
.enableHiveSupport() \
.getOrCreate()
with open('full/path/to/my/hqlfile') as t:
q=t.read()
print q
'use myDatabaseName;show tables;\n'
spark.sql(q)
但我明白了
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/some/path/python2.7/site-packages/pyspark/sql/session.py", line 767, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/some/path/python2.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/some/path/python2.7/site-packages/pyspark/sql/utils.py", line 73, in deco
raise ParseException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.ParseException: u"\nmismatched input ';' expecting <EOF>(line 1, pos 11)\n\n== SQL ==\nuse myDatabaseName;show tables;\n-----------^^^\n"
我做错什么了?
1条答案
按热度按时间a0x5cqrl1#
正如错误所暗示的,
;
不是spark.sql中的有效语法,其次,不能在一个spark.sql调用中调用两个命令。
我将修改
q
是一个没有;
在它里面然后循环。