运行中的spark,但yarn找不到它

kqlmhetl  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(797)

我有一个关于纱团的问题
我在localhost上运行hdfs namenode、hdfs datanode、yarn,然后在localhost上运行spark master和spark worker,如下所示:

$ jps  
5809 Main  
53730 ResourceManager  
53540 SecondaryNameNode  
53125 NameNode  
56710 Master  
54009 NodeManager  
56809 Worker  
53308 DataNode  
56911 Jps

我可以看到Spark工人是链接到Spark大师扔 http://127.0.0.1:8080 img:spark网络用户界面
[![在此处输入图像描述][1]][1]
但在网络用户界面中 http://127.0.0.1:8088 ,里面什么都没有 Nodes of the cluster 页码
图像:
[![在此处输入图像描述][2]][2]
我的 conf/spark-env.sh

export SCALA_HOME="/opt/scala-2.11.8/"  
export JAVA_HOME="/opt/jdk1.8.0_101/"  
export HADOOP_HOME="/opt/hadoop-2.7.3/"  
export HADOOP_CONF_DIR="/opt/hadoop-2.7.3/etc/hadoop/"  
export SPARK_MASTER_IP=127.0.0.1  
export SPARK_LOCAL_DIRS="/opt/spark-2.0.0-bin-hadoop2.7/"  
export SPARK_DRIVER_MEMORY=1G

以及 conf/spark-defaults.conf

spark.master                            spark://127.0.0.1:7077  
spark.yarn.submit.waitAppCompletion     false  
spark.yarn.access.namenodes             hdfs://127.0.0.1:8032

以及 yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>127.0.0.1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8035</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
</configuration>

当我提交申请时使用 spark-submit --master yarn --deploy-mode cluster test.py 我可以这样出去

16/10/12 16:19:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/12 16:19:30 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
16/10/12 16:19:30 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
16/10/12 16:19:30 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/10/12 16:19:30 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/10/12 16:19:30 INFO yarn.Client: Setting up container launch context for our AM
16/10/12 16:19:30 INFO yarn.Client: Setting up the launch environment for our AM container
16/10/12 16:19:30 INFO yarn.Client: Preparing resources for our AM container
16/10/12 16:19:31 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
16/10/12 16:19:32 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909/__spark_libs__2140674596658903486.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/__spark_libs__2140674596658903486.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/home/fuxiuyin/PycharmProjects/spark-test/test.py -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/test.py
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/pyspark.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/py4j-0.10.1-src.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909/__spark_conf__3570291475444079549.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/__spark_conf__.zip
16/10/12 16:19:33 INFO spark.SecurityManager: Changing view acls to: fuxiuyin
16/10/12 16:19:33 INFO spark.SecurityManager: Changing modify acls to: fuxiuyin
16/10/12 16:19:33 INFO spark.SecurityManager: Changing view acls groups to:
16/10/12 16:19:33 INFO spark.SecurityManager: Changing modify acls groups to:
16/10/12 16:19:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(fuxiuyin); groups with view permissions: Set(); users  with modify permissions: Set(fuxiuyin); groups with modify permissions: Set()
16/10/12 16:19:33 INFO yarn.Client: Submitting application application_1476256306830_0002 to ResourceManager
16/10/12 16:19:33 INFO impl.YarnClientImpl: Submitted application application_1476256306830_0002
16/10/12 16:19:33 INFO yarn.Client: Application report for application_1476256306830_0002 (state: ACCEPTED)
16/10/12 16:19:33 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1476260373944
     final status: UNDEFINED
     tracking URL: http://localhost:8088/proxy/application_1476256306830_0002/
     user: fuxiuyin
16/10/12 16:19:33 INFO util.ShutdownHookManager: Shutdown hook called
16/10/12 16:19:33 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909

虽然很成功,但在yarn web ui中,该应用程序并非总是以可接受的方式运行

似乎没有spark节点运行此应用程序。
有人能告诉我怎么了吗?
谢谢~

sycxhyv7

sycxhyv71#

可以指定一种类型的群集:
Yarn(集群或客户机模式)
spark单机版
中观
您已启动spark独立服务器,并且正在连接到此群集管理器。如果你想开始在Yarn上Spark,你应该指定 yarn 主人-只是 --master yarn 编辑:请添加日志和 spark-submit 命令。也请张贴你是如何推出纱。如果第一次尝试是错误的,那么这意味着您有配置问题
第二点:看起来yarn没有足够的资源来处理你的应用程序。请检查您的配置,即检查是否 yarn.nodemanager.resource.memory-mb 会有帮助的。你也可以去spark网页界面-http://application-master-ip:4040-查看spark上下文中的信息。
另外,只需设置 --master spark://...: 与配置相同。那你就可以确定是Yarn问题还是Spark问题了
顺便说一句,如果您提交给yarn:,您可以省略运行spark standalone,而由stanalone工作人员使用的内存可以由yarn使用

6uxekuva

6uxekuva2#

:). 谢谢大家,很抱歉浪费你们的时间。当我签入资源时 http://localhost:8088/ 我注意到:

我只是停止服务器并删除tmp目录和logs目录。那就行了。
再次感谢

相关问题