flink会话-请求的资源不可用

oxcyiej7  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(432)

我已经建立了一个hadoop2.7.5.ha集群，并使用默认的yarn队列运行flink1.4.0应用程序。我决定对应用程序进行分类并在独占节点管理器上运行它们，所以我标记了三个节点，每个节点 4 core 以及 2GB RAM 作为 stream 排队 streamQ 每个节点三个 1 core 以及 1GB RAM 作为 online 排队 onlineQ 所有设置都会根据需要显示在yarn webui中，并标识节点。这是你的名字 capacity-scheduler.xml :

<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
</property>

<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>
</property>

<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
</property>

<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>40</value>
</property>

<property>
<name>yarn.scheduler.capacity.queue-mappings</name>
<value></value>
</property>

<property>
<name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
<value>false</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>streamQ,onlineQ</value>
</property>

<!-- streamQ settings -->

<property>
<name>yarn.scheduler.capacity.root.streamQ.capacity</name>
<value>0</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels</name>
<value>stream</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels.stream.capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels.stream.maximum-capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.default-node-label-expression</name>
<value>stream</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.user-limit-factor</name>
<value>1</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.maximum-capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.state</name>
<value>RUNNING</value>
</property>

<property>

<name>yarn.scheduler.capacity.root.streamQ.acl_submit_applications</name>
<value>*</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.streamQ.acl_administer_queue</name>
<value>*</value>
</property>

<!-- onlineQ settings -->

<property>
<name>yarn.scheduler.capacity.root.onlineQ.capacity</name>
<value>0</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels</name>
<value>online</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels.online.capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels.online.maximum-capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.default-node-label-expression</name>
<value>online</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.user-limit-factor</name>
<value>1</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.maximum-capacity</name>
<value>100</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.state</name>
<value>RUNNING</value>
</property>

<property>
 <name>yarn.scheduler.capacity.root.onlineQ.acl_submit_applications</name>
<value>*</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.onlineQ.acl_administer_queue</name>
<value>*</value>
</property>

我运行命令在边缘节点上启动flink会话，所有hadoop配置与cluster相同：

yarn-session.sh -n 2 -jm 768 -tm 768 -nm flink -z flink_zoo -s 3 -qu streamQ

它成功地将flink libs上传到hdfs上，在yarn webui中我可以看到应用程序，但是当它试图获取资源时，它说：

018-01-28 10:02:04,087 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

以下是全部日志：

2018-01-28 10:00:09,648 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-01-28 10:00:09,649 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 768
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 768
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-01-28 10:00:09,650 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: web.port, 8081
2018-01-28 10:00:10,003 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-01-28 10:00:10,069 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to manager (auth:SIMPLE)
2018-01-28 10:00:10,377 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Cluster specification: ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=768, numberTaskManagers=2, slotsPerTaskManager=3}
2018-01-28 10:00:10,747 WARN  org.apache.flink.yarn.YarnClusterDescriptor                   - The configuration directory ('/opt/flink/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2018-01-28 10:00:10,751 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/conf/log4j.properties to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/log4j.properties
2018-01-28 10:00:11,123 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/log4j-1.2.17.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/lib/log4j-1.2.17.jar
2018-01-28 10:00:11,384 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/flink-dist_2.11-1.4.0.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/lib/flink-dist_2.11-1.4.0.jar
2018-01-28 10:00:30,986 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/flink-shaded-hadoop2-uber-1.4.0.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/lib/flink-shaded-hadoop2-uber-1.4.0.jar
2018-01-28 10:00:40,852 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/flink-python_2.11-1.4.0.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/lib/flink-python_2.11-1.4.0.jar
2018-01-28 10:00:41,017 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/slf4j-log4j12-1.7.7.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/lib/slf4j-log4j12-1.7.7.jar
2018-01-28 10:00:41,250 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/conf/logback.xml to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/logback.xml
2018-01-28 10:00:41,386 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/opt/flink/lib/flink-dist_2.11-1.4.0.jar to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/flink-dist_2.11-1.4.0.jar
2018-01-28 10:01:02,966 INFO  org.apache.flink.yarn.Utils                                   - Copying from /tmp/application_1517118829753_0002-flink-conf.yaml285707454205346702.tmp to hdfs://ha-cluster/user/manager/.flink/application_1517118829753_0002/application_1517118829753_0002-flink-conf.yaml285707454205346702.tmp
2018-01-28 10:01:03,601 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Submitting application master application_1517118829753_0002
2018-01-28 10:01:03,782 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1517118829753_0002
2018-01-28 10:01:03,783 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Waiting for the cluster to be allocated
2018-01-28 10:01:03,796 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Deploying cluster, current state ACCEPTED

有什么问题？

hadoopyarnapache-flink

2条答案

按热度按时间
798qvoo8

798qvoo81#

编辑 capacity-scheduler.xml ，解决了问题：

<!-- configuration of queue-root -->

<property> 
  <name>yarn.scheduler.capacity.root.queues</name> 
  <value>streamQ,onlineQ</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.accessible-node-labels</name> 
  <value>*</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.accessible-node-labels.stream.capacity</name> 
  <value>100</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.accessible-node-labels.online.capacity</name> 
  <value>100</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.default-node-label-expression</name> 
  <value>*</value> 
</property>

 <!-- configuration of queue-streamQ -->

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.capacity</name> 
  <value>50</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.maximum-capacity</name> 
  <value>100</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels</name> 
  <value>stream</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels.stream.capacity</name> 
  <value>100</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.accessible-node-labels.online.capacity</name> 
  <value>0</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.streamQ.default-node-label-expression</name> 
  <value>stream</value> 
</property>

<!-- configuration of queue-streamQ -->

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.capacity</name> 
  <value>50</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.maximum-capacity</name> 
  <value>100</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels</name> 
  <value>online</value> 
</property>

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels.online.capacity</name> 
  <value>100</value>
</property>

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.accessible-node-labels.stream.capacity</name> 
  <value>0</value>
</property>

<property> 
  <name>yarn.scheduler.capacity.root.onlineQ.default-node-label-expression</name> 
  <value>online</value> 
</property>

</configuration>
赞(0）分享  回复(0）举报  2021-05-29
oxiaedzo

oxiaedzo2#

请检查您的flink应用程序日志，查看连接到资源管理器时是否有问题。我也遇到了这个问题，当我使用flink对纱与ha。我不知道如果我是唯一的一个。

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com