livyclient uploadjar失败,py4j.py4jexception:获取新通信通道时出错

zazmityj  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(594)

我试图通过apachelivy提交spark作业,但是livyclient的uploadjar方法失败了。
这是代码(非常类似于pijob示例):

LivyClientBuilder builder = new LivyClientBuilder();
        LivyClient client = builder.setURI(new URI("http://server:8998")).build();
        client.uploadJar(new File("/path/to/file")).get();

以下是完整的堆栈跟踪:

py4j.Py4JException: Error while obtaining a new communication channel
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:257)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:377)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
    at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
    at com.sun.proxy.$Proxy24.getLocalTmpDirPath(Unknown Source)
    at org.apache.livy.repl.PythonInterpreter.addPyFile(PythonInterpreter.scala:294)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:114)
    at org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151)
    at org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:244)
    at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
    at py4j.CallbackConnection.start(CallbackConnection.java:226)
    at py4j.CallbackClient.getConnection(CallbackClient.java:238)
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:250)
    ... 17 more

我可以通过restapi向livy服务器提交代码片段,它们运行得很好。livy/spark是在yarn上设置的,我尝试了客户机和集群模式。有什么想法吗?

nkhmeac6

nkhmeac61#

从您的代码来看,您的jar路径似乎没有设置,请尝试设置jar的路径。
例子

client.uploadJar(new File("tmp/myjar.jar")).get();

您还可以检查pyspark环境的env路径,确保节点在同一环境中通信。

相关问题