kubernetes TransportChannelHandler:从/172.31.88.129:32691连接时发生异常java.lang.IllegalArgumentException:框架过大:5135603447296520

u4vypkhs  于 2023-05-16  发布在  Kubernetes
关注(0)|答案(1)|浏览(137)

我在使用大数据工具测试k8s集群中运行的spark时遇到了以下问题:

23/05/10 11:38:13 WARN TransportChannelHandler: Exception in connection from /172.31.88.129:32691
java.lang.IllegalArgumentException: Too large frame: 5135603447296520289
        at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
        at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

我使用nodeport暴露了spark 7077,它可以使用telnet进行外部连接。日志显示spark运行正常。我使用了相同的nodeport方式来公开sparkUI,使其可以正常访问enter image description here以下是大数据工具的错误消息:

com.jetbrains.bigdatatools.common.connection.exception.BdtRestExecutionException: Error during 'GET: /api/v1/version' request.
NoHttpResponseException: 172.31.88.129:30210 failed to respond
    at com.jetbrains.bigdatatools.common.connection.BdtRestExceptionHelper.wrapRest(BdtRestExceptionHelper.kt:27)
    at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.getVersion(SparkMonitoringRestClient.kt:127)
    at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.checkConnectionInner(SparkMonitoringRestClient.kt:20)
    at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.connectInner(SparkMonitoringRestClient.kt:30)
    at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient$connect$1.invoke(MonitoringClient.kt:32)
    at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient$connect$1.invoke(MonitoringClient.kt:31)
    at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connectWrapper(MonitoringClient.kt:77)
    at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connect(MonitoringClient.kt:31)
    at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connectWithThrowable(MonitoringClient.kt:36)
    at com.jetbrains.spark.monitoring.connection.SparkMonitoringConnectionChecker.checkConnection(SparkMonitoringConnectionChecker.kt:21)
    at com.jetbrains.spark.monitoring.settings.SparkConnectionTestingBase.checkConnection(SparkConnectionTestingBase.kt:21)
    at com.jetbrains.spark.monitoring.settings.SparkConnectionTestingBase.checkConnection(SparkConnectionTestingBase.kt:13)
    at com.jetbrains.bigdatatools.common.rfs.settings.RfsConnectionTestingBase.testConnection(RfsConnectionTestingBase.kt:22)
    at com.jetbrains.bigdatatools.common.settings.connections.ConnectionTesting.testConnection(ConnectionData.kt:119)
    at com.jetbrains.bigdatatools.common.settings.defaultui.TestConnectionPanelWrapper$runTesting$1.invoke(TestConnectionPanelWrapper.kt:105)
    at com.jetbrains.bigdatatools.common.settings.defaultui.TestConnectionPanelWrapper$runTesting$1.invoke(TestConnectionPanelWrapper.kt:84)
    at com.jetbrains.bigdatatools.common.util.ThreadUtilsKt.executeOnPooledThread$lambda$3(ThreadUtils.kt:31)
    at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:252)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.jetbrains.bigdatatools.common.connection.exception.impl.RestExecutionException: Error during 'GET: /api/v1/version' request.

那么这个问题怎么解决呢?谢谢
我试图使用其他端口连接到spark master,但失败了。

qhhrdooz

qhhrdooz1#

这可能与spark版本冲突有关。
您可以检查版本的兼容性,根据需要进行修改并重新测试。

相关问题