hadoop缺少containerlaunchcontext.setresource()

sr4lhrrt  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(261)

http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/writingyarnapplications.html
我正试图从上面的链接使这个例子很好地工作。但是我不能编译下面的代码

Resource capability = Records.newRecord(Resource.class);
 capability.setMemory(512);
 amContainer.setResource(capability);

 // Set the container launch content into the
 // ApplicationSubmissionContext
 appContext.setAMContainerSpec(amContainer);

amcontainer是containerlaunchcontext,我的hadoop版本是2.1.0-beta。我做了些调查。我发现containerlaunchcontext中没有方法“setresource”
关于这个我有三个问题
1) 方法被删除了还是什么?
2) 如果该方法已被删除,我现在该怎么办?
3) 有没有关于Yarn的文件,因为我发现网站上的文件很容易,我希望我能得到一个手册或什么的。例如,capability.setmemory(512);我不知道这是512k或512m根据代码中的注解。

cyvaqqii

cyvaqqii1#

这实际上是解决这个问题的正确方法。前面的答案可能会导致错误的执行!!!
@因为我无法把它放在评论中;)已针对2.2.0和2.3.0进行验证
为appmaster设置资源的驱动程序:

ApplicationSubmissionContext appContext = app.getApplicationSubmissionContext();
ApplicationId appId = appContext.getApplicationId();
appContext.setApplicationName(this.appName);

// Set up the container launch context for the application master
ContainerLaunchContext amContainer = Records.newRecord(ContainerLaunchContext.class);

Resource capability = Records.newRecord(Resource.class);
capability.setMemory(amMemory);        
appContext.setResource(capability);

appContext.setAMContainerSpec(amContainer);

Priority pri = Records.newRecord(Priority.class);
pri.setPriority(amPriority);
appContext.setPriority(pri);

appContext.setQueue(amQueue);

// Submit the application to the applications manager
yarnClient.submitApplication(appContext); // this.yarnClient = YarnClient.createYarnClient();

在applicationmaster中,这是您应该如何为容器(worker)指定资源的。

private AMRMClient.ContainerRequest setupContainerAskForRM() {
        // setup requirements for hosts
        // using * as any host will do for the distributed shell app
        // set the priority for the request
        Priority pri = Records.newRecord(Priority.class);
        pri.setPriority(requestPriority);

        // Set up resource type requirements
        // For now, only memory is supported so we set memory requirements
        Resource capability = Records.newRecord(Resource.class);
        capability.setMemory(containerMemory);

        AMRMClient.ContainerRequest request = new AMRMClient.ContainerRequest(capability, null, null,
                pri);
        return request;
    }

appmaster中的一些run()或main()方法

AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
resourceManager = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
resourceManager.init(conf);
resourceManager.start();

for (int i = 0; i < numTotalContainers; ++i) {
  AMRMClient.ContainerRequest containerAsk = setupContainerAskForRM();
  resourceManager.addContainerRequest(containerAsk); // 
}

启动容器您可以使用原始的答案解决方案(javacmd),但它只是一个樱桃在上面。不管怎样都应该管用。

tct7dpnv

tct7dpnv2#

您可以通过命令设置applicationmaster可用的内存。像这样的:

// Set the necessary command to execute the application master
Vector<CharSequence> vargs = new Vector<CharSequence>(30);
...
vargs.add("-Xmx" + amMemory + "m"); // notice "m" indicating megabytes, you can use also -Xms combined with -Xmx
... // transform vargs to String commands
amContainer.setCommands(commands);

这应该能解决你的问题。至于这三个问题。Yarn是快速发展的软件。我的建议是忘记文档,获取源代码并阅读它。这将回答你的许多问题。

相关问题