设置:
节点数:3
芯数:每台机器32芯
内存:每台机器410gb
spark版本:1.2.0
hadoop版本:2.4.0(hortonworks)
目标:
我想运行一个超过32个执行器内核的spark作业。
问题:
当我为spark作业请求超过32个executor Core时,我得到以下错误:
Uncaught exception: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=150, maxVirtualCores=32
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:212)
at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:96)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:501)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
以下是myyarn-site.xml的一些属性:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>400000</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>3072</value>
</property>
从上面到属性,我想我可以要求400k/3072=~130核。但它仍然把我限制在32岁。如何将32个以上的执行器内核分配给spark作业?
请让我知道如果需要更多的信息,我会更新问题。
编辑1:
warn-site.xml中的vcore设置
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
1条答案
按热度按时间fhg3lkii1#
在yarn-site.xml集合中