spark-submit --packages com.databricks:spark-redshift_2.11:2.0.1 --jars /usr/share/aws/redshift/jdbc/RedshiftJDBC4.jar /home/hadoop/test.py
如何以apachelivy格式指定上述(pyspark)spark submit命令?
我尝试了以下方法:
curl -X POST --data '{"file": "/home/hadoop/test.py", "conf":
{"com.databricks": "spark-redshift_2.11:2.0.1"}, \
"queue": "my_queue", "name": "Livy Example", "jars" :
"/usr/share/aws/redshift/jdbc/RedshiftJDBC4.jar"}', \
-H "Content-Type: application/json" localhost:8998/batches
参考了下面的livy文章spark livy restapi
同时出现以下错误:
"Unexpected character ('“' (code 8220 / 0x201c)): was expecting double-quote to start field name\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 37]
1条答案
按热度按时间vltsax251#
您的命令错误,请使用下面的示例来构造命令。
spark提交命令
livy rest json协议
https://community.hortonworks.com/articles/151164/how-to-submit-spark-application-through-livy-rest.html
https://dzone.com/articles/quick-start-with-apache-livy
--包裹。使用此命令时,将处理所有可传递的依赖项。
在livy中,您需要转到解释器设置页,并在livy设置下添加新属性-
livy.spark.jars. Package
价值呢
重新启动解释器并重试查询。