我有一个运行在minikube
中的spark集群:
$ k get pods
NAME READY STATUS RESTARTS AGE
spark-master-0 1/1 Running 0 7d
spark-worker-0 1/1 Running 0 7d
spark-worker-1 1/1 Running 0 7d
我已将端口转发给master:
$ k port-forward spark-master-0 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
我正在尝试创建一个JavaSparkSession
,但我不确定要使用哪个主URL:
SparkConf sparkConf = new SparkConf()
.setMaster("???")
.setAppName("test-spark");
SparkSession.builder()
.config(sparkConf)
.getOrCreate();
已尝试以下方法,但未成功
"local"
"localhost:8080
http://localhost:8080
spark://localhost:8080
- (来自其中一个答案)
k8s://http://192.168.1.2:8080
- (来自其中一个答案)
k8s://https://kubernetes:443
它们都不起作用。得到以下错误:
Caused by: org.apache.spark.SparkException: Could not parse Master URL: '<MASTER URL FROM LIST ABOVE>'
我用什么主URL?谢谢
EDITUrl spark://192.168.58.2:7077
部分工作,出现如下错误:
2023-04-12T14:53:50.220-07:00 INFO 40229 --- [ main] o.s.jetty.server.handler.ContextHandler : Started o.s.j.s.ServletContextHandler@49c1e294{/,null,AVAILABLE,@Spark}
2023-04-12T14:53:50.255-07:00 INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:10.265-07:00 INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:30.264-07:00 INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:50.272-07:00 WARN 40229 --- [ main] o.a.s.s.c.StandaloneSchedulerBackend : Application ID is not initialized yet.
2023-04-12T14:54:50.272-07:00 ERROR 40229 --- [on-retry-thread] o.a.s.s.c.StandaloneSchedulerBackend : Application has been killed. Reason: All masters are unresponsive! Giving up.
2023-04-12T14:54:50.284-07:00 INFO 40229 --- [p-spark-context] o.s.jetty.server.AbstractConnector : Stopped Spark@7b6141f0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
2023-04-12T14:54:50.286-07:00 INFO 40229 --- [p-spark-context] org.apache.spark.ui.SparkUI : Stopped Spark web UI at http://mgc14dlpj2f2.lan:4040
2023-04-12T14:54:50.289-07:00 INFO 40229 --- [p-spark-context] o.a.s.s.c.StandaloneSchedulerBackend : Shutting down all executors
...
java.lang.NullPointerException: Cannot invoke "org.apache.spark.rpc.RpcEndpointRef.askSync(Object, scala.reflect.ClassTag)" because the return value of "org.apache.spark.storage.BlockManagerMaster.driverEndpoint()" is null
...
1条答案
按热度按时间j91ykkif1#
这应该可以工作:
**更新:**我与K8s,在你的情况下,它应该是Spark,我也检查了,我不使用HTTPS后k8s