所以我们对容量调度器还不熟悉。我们正在构建一个aws集群,希望在其中添加容量调度器配置,以使作业在不同的队列中同时运行。
问题是,尽管我们正设法使用scheduler config创建一个稳定的集群,但是我们无法在每个队列中并行提交作业。
通过参考下面的链接,我们通过提供相应的值来创建配置。https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/capacityscheduler.html
在上述链接中添加了所有参数后,集群在引导时会失败。
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=yarn
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=yarn
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=2
yarn.scheduler.capacity.root.queues=bt,default,opt
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.bt.acl_administer_queue=*
yarn.scheduler.capacity.root.bt.acl_submit_applications=*
yarn.scheduler.capacity.root.bt.capacity=25
yarn.scheduler.capacity.root.bt.maximum-capacity=100
yarn.scheduler.capacity.root.bt.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.bt.ordering-policy=fair
yarn.scheduler.capacity.root.bt.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.bt.priority=0
yarn.scheduler.capacity.root.bt.state=RUNNING
yarn.scheduler.capacity.root.bt.user-limit-factor=1
yarn.scheduler.capacity.root.default.acl_administer_queue=yarn
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=25
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.opt.acl_administer_queue=*
yarn.scheduler.capacity.root.opt.acl_submit_applications=*
yarn.scheduler.capacity.root.opt.capacity=25
yarn.scheduler.capacity.root.opt.maximum-capacity=25
yarn.scheduler.capacity.root.opt.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.opt.ordering-policy=fair
yarn.scheduler.capacity.root.opt.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.opt.priority=0
yarn.scheduler.capacity.root.opt.state=RUNNING
yarn.scheduler.capacity.root.opt.user-limit-factor=1
yarn.scheduler.capacity.root.priority=0
通过上述配置,我们可以创建一个集群,但我们面临的一些问题是:1。作业在默认队列中运行得非常好,如果在其他队列中提交,它将停留在接受状态。2一次只提交一个作业,其他作业仍在emr步骤中等待,而不是在不同的队列中运行。
p、 s:我们提交给emrs的作业是由lambda函数触发的spark作业。
暂无答案!
目前还没有任何答案,快来回答吧!