lib到livy交互会话

hrysbysz  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(325)

我有图书馆（这是链接），我去添加到我的livy互动会话。为此，我将其添加到spark-defaults.conf中，如下所示：

spark.jars.repositories https://dl.bintray.com/unsupervise/maven
spark.jars.packages com.github.unsupervise:spark-tss:0.1.1

当我添加新的livy会话时，出现以下错误，会话状态为dead：

20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 WARN RSCDriver: Error during cancel job.
20/06/12 01:52:50 INFO utils.LineBufferedStream: java.lang.NullPointerException
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:128)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:360)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.lang.reflect.Method.invoke(Method.java:498)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/06/12 01:52:50 WARN rsc.RSCClient: Client RPC channel closed unexpectedly.
20/06/12 01:52:50 WARN rsc.RSCClient: Error stopping RPC.
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@1fc443fe(uncancellable)
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
        at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:230)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
        at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:336)
        at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)
        at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
        at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:310)
        at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:235)
        at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:132)
        at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:126)
        at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:106)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:492)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:995)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:621)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:599)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:543)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
        at java.lang.Thread.run(Thread.java:748)
20/06/12 01:52:50 INFO rsc.RSCClient: Failing pending job d7b2b9e5-b67d-4ce8-8f22-fa66304d0997 due to shutdown.
20/06/12 01:52:50 INFO interactive.InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
20/06/12 01:52:50 INFO interactive.InteractiveSession: Stopping InteractiveSession 0...
20/06/12 01:52:50 INFO utils.LineBufferedStream: Exception in thread "main" scala.reflect.internal.FatalError: object Predef does not have a member classOf
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1182)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1199)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1234)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1465)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1465)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1455)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4912)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4935)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5367)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5039)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5339)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5386)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5477)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:467)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:458)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1528)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1513)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.Global$Run.compileSources(Global.scala:1508)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:442)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:862)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:820)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:682)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter$$anonfun$bind$1.apply(SparkInterpreter.scala:132)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter$$anonfun$bind$1.apply(SparkInterpreter.scala:132)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter.bind(SparkInterpreter.scala:131)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:72)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:88)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:63)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:63)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.lang.Thread.run(Thread.java:748)
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 INFO SparkContext: Invoking stop() from shutdown hook
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 INFO SparkUI: Stopped Spark web UI at http://mycluster-master:4040
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 WARN Client: interrupted waiting to send rpc request to server
20/06/12 01:52:50 INFO utils.LineBufferedStream: java.lang.InterruptedException
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.FutureTask.get(FutureTask.java:191)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client.call(Client.java:1454)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at com.sun.proxy.$Proxy13.delete(Unknown Source)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:540)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.lang.reflect.Method.invoke(Method.java:498)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at com.sun.proxy.$Proxy14.delete(Unknown Source)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$cleanupStagingDirInternal$1(Client.scala:209)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.cleanupStagingDir(Client.scala:226)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1090)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:114)
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 WARN Client: Failed to cleanup staging dir hdfs://mycluster-master:9000/user/root/.sparkStaging/application_1591892001338_0007
20/06/12 01:52:50 INFO utils.LineBufferedStream: java.io.IOException: java.lang.InterruptedException
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client.call(Client.java:1460)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at com.sun.proxy.$Proxy13.delete(Unknown Source)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:540)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.lang.reflect.Method.invoke(Method.java:498)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at com.sun.proxy.$Proxy14.delete(Unknown Source)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$cleanupStagingDirInternal$1(Client.scala:209)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.cleanupStagingDir(Client.scala:226)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1090)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:114)
20/06/12 01:52:50 INFO utils.LineBufferedStream: Caused by: java.lang.InterruptedException
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at java.util.concurrent.FutureTask.get(FutureTask.java:191)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        at org.apache.hadoop.ipc.Client.call(Client.java:1454)
20/06/12 01:52:50 INFO utils.LineBufferedStream:        ... 20 more
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state KILLED! Check the YARN application logs for more details.
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 ERROR YarnClientSchedulerBackend: Diagnostics message: Application killed by user.
20/06/12 01:52:50 INFO utils.LineBufferedStream: 20/06/12 01:52:50 INFO SparkContext: SparkContext already stopped.
20/06/12 01:52:50 INFO impl.YarnClientImpl: Killed application application_1591892001338_0007
20/06/12 01:52:50 INFO interactive.InteractiveSession: Stopped InteractiveSession 0.
Exception in thread "Thread-33" java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
20/06/12 01:52:51 WARN rsc.ContextLauncher: Child process exited with code 1.

我使用：
利维0.7.0
Spark2.4.1
hadoop 2.7.7版
斯卡拉2.11.12
livy.spark.master=Yarn
livy.spark.deploy-mode=客户端
注意：当我从spark-defaults.conf中删除lib时，一切都很好

scalaapache-sparklivy

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com