我试图通过apachelivy提交spark作业,但是livyclient的uploadjar方法失败了。
这是代码(非常类似于pijob示例):
LivyClientBuilder builder = new LivyClientBuilder();
LivyClient client = builder.setURI(new URI("http://server:8998")).build();
client.uploadJar(new File("/path/to/file")).get();
以下是完整的堆栈跟踪:
py4j.Py4JException: Error while obtaining a new communication channel
at py4j.CallbackClient.getConnectionLock(CallbackClient.java:257)
at py4j.CallbackClient.sendCommand(CallbackClient.java:377)
at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
at com.sun.proxy.$Proxy24.getLocalTmpDirPath(Unknown Source)
at org.apache.livy.repl.PythonInterpreter.addPyFile(PythonInterpreter.scala:294)
at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
at scala.Option.foreach(Option.scala:257)
at org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:114)
at org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151)
at org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39)
at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
at py4j.CallbackConnection.start(CallbackConnection.java:226)
at py4j.CallbackClient.getConnection(CallbackClient.java:238)
at py4j.CallbackClient.getConnectionLock(CallbackClient.java:250)
... 17 more
我可以通过restapi向livy服务器提交代码片段,它们运行得很好。livy/spark是在yarn上设置的,我尝试了客户机和集群模式。有什么想法吗?
1条答案
按热度按时间nkhmeac61#
从您的代码来看,您的jar路径似乎没有设置,请尝试设置jar的路径。
例子
您还可以检查pyspark环境的env路径,确保节点在同一环境中通信。