HDFS Haddop集群有3个活动节点,但只有1个节点在资源管理器中处于活动状态

vsaztqbk  于 2023-10-14  发布在  HDFS
关注(0)|答案(1)|浏览(253)

我在docker上运行hadoop,使用docker hub的apache/hadoop镜像。
我只是运行与默认配置和添加2模式datanode
在这里我的码头-组成。yaml

version: "2"
services:
   namenode:
      image: apache/hadoop:3.3.6
      hostname: namenode0
      command: ["hdfs", "namenode"]
      ports:
        - 9870:9870
      env_file:
        - ./config
      environment:
          ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
   datanode-0:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   datanode-1:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   datanode-2:
      image: apache/hadoop:3.3.6
      command: ["hdfs", "datanode"]
      env_file:
        - ./config
   resourcemanager:
      image: apache/hadoop:3.3.6
      hostname: resourcemanager
      command: ["yarn", "resourcemanager"]
      ports:
         - 8088:8088
      env_file:
        - ./config
      volumes:
        - ./test.sh:/opt/test.sh
   nodemanager:
      image: apache/hadoop:3.3.6
      command: ["yarn", "nodemanager"]
      env_file:
        - ./config

但在namenode UI中有3个活动节点x1c 0d1x
如果运行这个

$ hdfs dfsadmin -report

Configured Capacity: 763068678144 (710.66 GB)
Present Capacity: 405373390848 (377.53 GB)
DFS Remaining: 405373366272 (377.53 GB)
DFS Used: 24576 (24 KB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
Erasure Coded Block Groups: 
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 172.25.0.4:9866 (hadoop_datanode-0_1.hadoop_default)
Hostname: 7eb5f69ae955
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:14 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0

Name: 172.25.0.5:9866 (hadoop_datanode-1_1.hadoop_default)
Hostname: 29d464fd39ac
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:13 UTC 2023
Last Block Report: Sat Oct 07 15:21:37 UTC 2023
Num of Blocks: 0

Name: 172.25.0.7:9866 (hadoop_datanode-2_1.hadoop_default)
Hostname: dccb08fe06b8
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:12 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0

但是在yarn和资源管理器中只有一个节点active

$ yarn node -list

2023-10-07 15:47:05 INFO  DefaultNoHARMFailoverProxyProvider:64 - Connecting to ResourceManager at resourcemanager/172.25.0.6:8032
Total Nodes:1
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
c8ac8dd71a0d:41135              RUNNING c8ac8dd71a0d:8042                                  0

如何将其他节点添加到主动节点?

4dbbbstv

4dbbbstv1#

Datanodes和Nodemanager是两个不同的东西。添加更多Nodemanager以更新YARN
请记住,您的系统可能最终耗尽内存试图运行所有这些容器,并没有真正的意义,因为它仍然是一个单点故障(你的主机)和竞争的同一组资源(CPU,硬盘和RAM),所以所有这些容器可能比在本地安装Hadoop并让它完全访问您的系统要慢,而不是强迫它在同一个硬盘驱动器内复制数据并从同一个主机进行内部(“假远程”)网络调用

相关问题