hadoop mapreduce作业因uxservice:mapreduce_shuffle 不存在

7kqas0il  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(461)

我已经检查了多篇带有相同问题的帖子,解决方案总是在yarn-site.xml中添加以下内容

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarm.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

我在config和restarted中介绍了这两个属性。问题仍然存在。
错误是:

17/02/15 15:43:34 INFO mapreduce.Job: Task Id : attempt_1487202110321_0001_m_000000_2, Status : FAILED
Container launch failed for container_1487202110321_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我本想找个打字错误,但似乎找不到。尝试直接将stackoverflow上的内容复制到xml文件中,仍然不起作用。
我还能试什么?
编辑:
因为错误说辅助设备服务应该 auxService ,我修改了yarn-site.xml,修改了 aux-serviceauxService ,但仍然不起作用。
编辑2:
如果有人感兴趣,我就叫这个命令

hadoop jar hadoop-streaming-2.7.1.jar \
       -input /user/myfolder/input1/* \
       -output /user/myfolder/output1 \
       -mapper <path>/<to>/<mapper>/mapper.py \
       -reducer <path>/<to>/<reducer>/reducer.py

当我已经在的时候 /usr/local/cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/ 编辑3:
我是个笨蛋。校对剧本伙计们!

tag5nh1u

tag5nh1u1#

更新中的属性名称 yarn-site.xml 作为 yarn.nodemanager.aux-services ,

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

相关问题