在Kubernetes中运行的Spark使用什么主URL?

qco9c6ql  于 2023-04-20  发布在  Kubernetes
关注(0)|答案(1)|浏览(137)

我有一个运行在minikube中的spark集群:

$ k get pods                                                                                                                                              
NAME             READY   STATUS    RESTARTS   AGE
spark-master-0   1/1     Running   0          7d
spark-worker-0   1/1     Running   0          7d
spark-worker-1   1/1     Running   0          7d

我已将端口转发给master:

$ k port-forward spark-master-0 8080:8080                                                                                                                   
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

我正在尝试创建一个JavaSparkSession,但我不确定要使用哪个主URL:

SparkConf sparkConf = new SparkConf()
    .setMaster("???")
    .setAppName("test-spark");
SparkSession.builder()
    .config(sparkConf)
    .getOrCreate();

已尝试以下方法,但未成功

  • "local"
  • "localhost:8080
  • http://localhost:8080
  • spark://localhost:8080
  • (来自其中一个答案)k8s://http://192.168.1.2:8080
  • (来自其中一个答案)k8s://https://kubernetes:443

它们都不起作用。得到以下错误:

Caused by: org.apache.spark.SparkException: Could not parse Master URL: '<MASTER URL FROM LIST ABOVE>'

我用什么主URL?谢谢

EDITUrl spark://192.168.58.2:7077部分工作,出现如下错误:

2023-04-12T14:53:50.220-07:00  INFO 40229 --- [           main] o.s.jetty.server.handler.ContextHandler  : Started o.s.j.s.ServletContextHandler@49c1e294{/,null,AVAILABLE,@Spark}
2023-04-12T14:53:50.255-07:00  INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:10.265-07:00  INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:30.264-07:00  INFO 40229 --- [er-threadpool-0] s.d.c.StandaloneAppClient$ClientEndpoint : Connecting to master spark://192.168.58.2:7077...
2023-04-12T14:54:50.272-07:00  WARN 40229 --- [           main] o.a.s.s.c.StandaloneSchedulerBackend     : Application ID is not initialized yet.
2023-04-12T14:54:50.272-07:00 ERROR 40229 --- [on-retry-thread] o.a.s.s.c.StandaloneSchedulerBackend     : Application has been killed. Reason: All masters are unresponsive! Giving up.
2023-04-12T14:54:50.284-07:00  INFO 40229 --- [p-spark-context] o.s.jetty.server.AbstractConnector       : Stopped Spark@7b6141f0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
2023-04-12T14:54:50.286-07:00  INFO 40229 --- [p-spark-context] org.apache.spark.ui.SparkUI              : Stopped Spark web UI at http://mgc14dlpj2f2.lan:4040
2023-04-12T14:54:50.289-07:00  INFO 40229 --- [p-spark-context] o.a.s.s.c.StandaloneSchedulerBackend     : Shutting down all executors
...
java.lang.NullPointerException: Cannot invoke "org.apache.spark.rpc.RpcEndpointRef.askSync(Object, scala.reflect.ClassTag)" because the return value of "org.apache.spark.storage.BlockManagerMaster.driverEndpoint()" is null
...
j91ykkif

j91ykkif1#

这应该可以工作:

k8s://https://{MasterIp}:{Port}

**更新:**我与K8s,在你的情况下,它应该是Spark,我也检查了,我不使用HTTPS后k8s

SparkConf conf = new SparkConf().setAppName("App_Name")
    .setMaster("spark://192.168.58.2:8080").set("spark.ui.port","8080");

相关问题