我在测试hadoop多节点集群时遇到了一些问题

d7v8vwbk  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(413)

嗨，我正在使用hadoop0.23.9与2节点当我尝试单节点工作正常，但当我尝试测试字计数工作，我得到以下结果，我不知道哪里的错误
我在两台电脑上都使用ubuntu 12.04 lts

mapred-site.xml

        <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        </property>

        </configuration>

         yarn-site.xml

<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>

 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  </property>

<property>
 <name>mapreduce.job.tracker</name>
  <value>master</value>
 </property>
<configuration>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>

<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>

<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8042</value>
</property>

</configuration>

这里是ubuntu的hosts文件

127.0.0.1       localhost
    127.0.1.1       fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

# the following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

        14/04/25 15:41:39 INFO input.FileInputFormat: Total input paths to process : 7
        14/04/25 15:41:39 WARN snappy.LoadSnappy: Snappy native library not loaded
        14/04/25 15:41:39 INFO mapreduce.JobSubmitter: number of splits:7
        14/04/25 15:41:39 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.value.class is deprecated.    Instead, use mapreduce.job.output.value.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
        14/04/25 15:41:39 INFO mapred.ResourceMgrDelegate: Submitted application application_1398429615167_0001 to ResourceManager at master/192.168.1.10:8032
        14/04/25 15:41:39 INFO mapreduce.Job: The url to track the job: `http://master:8088/proxy/application_1398429615167_0001/`
        14/04/25 15:41:39 INFO mapreduce.Job: Running job: job_1398429615167_0001
        14/04/25 15:41:46 INFO mapreduce.Job: Job job_1398429615167_0001 running in uber mode : false
        14/04/25 15:41:46 INFO mapreduce.Job:  map 0% reduce 0%
        14/04/25 15:41:47 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_0, Status : FAILED
    Container launch failed for container_1398429615167_0001_01_000008 : java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
    Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
        at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:49 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_1, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000009 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:51 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_2, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000010 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:53 INFO mapreduce.Job:  map 14% reduce 0%
14/04/25 15:41:53 INFO mapreduce.Job: Job job_1398429615167_0001 failed with state FAILED due to: Task failed task_1398429615167_0001_m_000006
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/04/25 15:41:53 INFO mapreduce.Job: Counters: 5
    Job Counters 
        Launched map tasks=6
        Other local map tasks=3
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0
hadoopmapreduceyarnapache

2条答案

按热度按时间
js4nwp54

js4nwp541#

在尝试了所有可能的解决方案之后，解决方案是：将pc的主机名设为“主”和“从”。然后在文件“/etc/hosts”中输入PC的ip地址。
非常感谢阿佩萨先生的回答。

赞(0）分享  回复(0）举报  2021-06-03
6qfn3psc

6qfn3psc2#

您的主机名在/etc/hosts中设置不正确。你应该把下面这一行全部删除。

127.0.1.1       fatima1-HP-ProBook-4530s

然后在localhost条目之后添加您的实际ip地址。您的/etc/hosts文件应该如下所示。注：我假设这是主，我可能是错误的，在这种情况下，取代它的实际ip地址。还要检查目标machines/etc/hosts文件。

127.0.0.1       localhost
    192.168.1.10    fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

完成后，您需要重新启动计算机并重新启动hadoop
编辑：core-site.xml必须有正确的主/从条目，如下所示。您可以在core-site.xml中设置其他属性，这很好。但在分布式模式下，您需要将fs.default.name指定为主或从。

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
</configuration>
赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com