hadoop 3.0/1的gpu资源

hgb9j2n6  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(544)

我尝试将hadoop3.0ga版本与gpu一起使用,但是当我执行below shell命令时,出现了一个错误,无法与gpu一起使用。请检查下面的命令并告诉您shell命令。我想我有误解。

2018-01-09 15:04:49,256 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:main(355)) - Initializing ApplicationMaster
2018-01-09 15:04:49,391 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:init(514)) - Application master for app, appId=1, clustertimestamp=1515477741976, attemptId=1
2018-01-09 15:04:49,418 WARN  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:init(626)) - Timeline service is not enabled
2018-01-09 15:04:49,418 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(649)) - Starting ApplicationMaster
2018-01-09 15:04:49,542 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-01-09 15:04:49,623 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(659)) - Executing with tokens:
2018-01-09 15:04:49,744 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(662)) - Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1 cluster_timestamp: 1515477741976 } attemptId: 1 } keyId: 1619387150)
2018-01-09 15:04:49,801 INFO  [main] client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at /0.0.0.0:8030
2018-01-09 15:04:49,886 INFO  [main] impl.NMClientAsyncImpl (NMClientAsyncImpl.java:serviceInit(138)) - Upper bound of the thread pool size is 500
2018-01-09 15:04:49,889 WARN  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(786)) - Timeline service is not enabled
2018-01-09 15:04:50,170 INFO  [main] conf.Configuration (Configuration.java:getConfResourceAsInputStream(2656)) - resource-types.xml not found
2018-01-09 15:04:50,170 INFO  [main] resource.ResourceUtils (ResourceUtils.java:addResourcesFileToConf(395)) - Unable to find 'resource-types.xml'.
2018-01-09 15:04:50,183 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,185 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,185 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,185 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,187 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,187 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,188 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,188 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,188 WARN  [main] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:50,188 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(717)) - Max mem capability of resources in this cluster 8192
2018-01-09 15:04:50,188 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(720)) - Max vcores capability of resources in this cluster 4
2018-01-09 15:04:50,189 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:run(739)) - appattempt_1515477741976_0001_000001 received 0 previous attempts' running containers on AM registration.
2018-01-09 15:04:50,202 INFO  [main] distributedshell.ApplicationMaster (ApplicationMaster.java:setupContainerAskForRM(1311)) - Requested container ask: Capability[<memory:-1, vCores:-1>]Priority[0]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[gpu-1]
2018-01-09 15:04:50,246 WARN  [AMRM Heartbeater thread] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:51,255 WARN  [AMRM Heartbeater thread] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:52,273 WARN  [AMRM Heartbeater thread] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping
2018-01-09 15:04:52,278 INFO  [AMRM Callback Handler Thread] distributedshell.ApplicationMaster (ApplicationMaster.java:onContainersAllocated(957)) - Got response from RM for container ask, allocatedCnt=1
2018-01-09 15:04:52,278 WARN  [AMRM Callback Handler Thread] pb.ResourcePBImpl (ResourcePBImpl.java:initResources(142)) - Got unknown resource type: yarn.io/gpu; skipping

我对yarn-7223票据执行的shell命令后面是,

yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \ -shell_command /usr/local/nvidia/bin/nvidia-smi -container_resource_profile gpu-1

提前谢谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题