我在使用大数据工具测试k8s集群中运行的spark时遇到了以下问题:
23/05/10 11:38:13 WARN TransportChannelHandler: Exception in connection from /172.31.88.129:32691
java.lang.IllegalArgumentException: Too large frame: 5135603447296520289
at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
我使用nodeport暴露了spark 7077,它可以使用telnet进行外部连接。日志显示spark运行正常。我使用了相同的nodeport方式来公开sparkUI,使其可以正常访问enter image description here以下是大数据工具的错误消息:
com.jetbrains.bigdatatools.common.connection.exception.BdtRestExecutionException: Error during 'GET: /api/v1/version' request.
NoHttpResponseException: 172.31.88.129:30210 failed to respond
at com.jetbrains.bigdatatools.common.connection.BdtRestExceptionHelper.wrapRest(BdtRestExceptionHelper.kt:27)
at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.getVersion(SparkMonitoringRestClient.kt:127)
at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.checkConnectionInner(SparkMonitoringRestClient.kt:20)
at com.jetbrains.spark.monitoring.connection.SparkMonitoringRestClient.connectInner(SparkMonitoringRestClient.kt:30)
at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient$connect$1.invoke(MonitoringClient.kt:32)
at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient$connect$1.invoke(MonitoringClient.kt:31)
at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connectWrapper(MonitoringClient.kt:77)
at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connect(MonitoringClient.kt:31)
at com.jetbrains.bigdatatools.common.monitoring.connection.MonitoringClient.connectWithThrowable(MonitoringClient.kt:36)
at com.jetbrains.spark.monitoring.connection.SparkMonitoringConnectionChecker.checkConnection(SparkMonitoringConnectionChecker.kt:21)
at com.jetbrains.spark.monitoring.settings.SparkConnectionTestingBase.checkConnection(SparkConnectionTestingBase.kt:21)
at com.jetbrains.spark.monitoring.settings.SparkConnectionTestingBase.checkConnection(SparkConnectionTestingBase.kt:13)
at com.jetbrains.bigdatatools.common.rfs.settings.RfsConnectionTestingBase.testConnection(RfsConnectionTestingBase.kt:22)
at com.jetbrains.bigdatatools.common.settings.connections.ConnectionTesting.testConnection(ConnectionData.kt:119)
at com.jetbrains.bigdatatools.common.settings.defaultui.TestConnectionPanelWrapper$runTesting$1.invoke(TestConnectionPanelWrapper.kt:105)
at com.jetbrains.bigdatatools.common.settings.defaultui.TestConnectionPanelWrapper$runTesting$1.invoke(TestConnectionPanelWrapper.kt:84)
at com.jetbrains.bigdatatools.common.util.ThreadUtilsKt.executeOnPooledThread$lambda$3(ThreadUtils.kt:31)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:252)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.jetbrains.bigdatatools.common.connection.exception.impl.RestExecutionException: Error during 'GET: /api/v1/version' request.
那么这个问题怎么解决呢?谢谢
我试图使用其他端口连接到spark master,但失败了。
1条答案
按热度按时间qhhrdooz1#
这可能与spark版本冲突有关。
您可以检查版本的兼容性,根据需要进行修改并重新测试。