我在docker上运行hadoop,使用docker hub的apache/hadoop镜像。
我只是运行与默认配置和添加2模式datanode
在这里我的码头-组成。yaml
version: "2"
services:
namenode:
image: apache/hadoop:3.3.6
hostname: namenode0
command: ["hdfs", "namenode"]
ports:
- 9870:9870
env_file:
- ./config
environment:
ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
datanode-0:
image: apache/hadoop:3.3.6
command: ["hdfs", "datanode"]
env_file:
- ./config
datanode-1:
image: apache/hadoop:3.3.6
command: ["hdfs", "datanode"]
env_file:
- ./config
datanode-2:
image: apache/hadoop:3.3.6
command: ["hdfs", "datanode"]
env_file:
- ./config
resourcemanager:
image: apache/hadoop:3.3.6
hostname: resourcemanager
command: ["yarn", "resourcemanager"]
ports:
- 8088:8088
env_file:
- ./config
volumes:
- ./test.sh:/opt/test.sh
nodemanager:
image: apache/hadoop:3.3.6
command: ["yarn", "nodemanager"]
env_file:
- ./config
但在namenode UI中有3个活动节点x1c 0d1x
如果运行这个
$ hdfs dfsadmin -report
Configured Capacity: 763068678144 (710.66 GB)
Present Capacity: 405373390848 (377.53 GB)
DFS Remaining: 405373366272 (377.53 GB)
DFS Used: 24576 (24 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 172.25.0.4:9866 (hadoop_datanode-0_1.hadoop_default)
Hostname: 7eb5f69ae955
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:14 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0
Name: 172.25.0.5:9866 (hadoop_datanode-1_1.hadoop_default)
Hostname: 29d464fd39ac
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:13 UTC 2023
Last Block Report: Sat Oct 07 15:21:37 UTC 2023
Num of Blocks: 0
Name: 172.25.0.7:9866 (hadoop_datanode-2_1.hadoop_default)
Hostname: dccb08fe06b8
Decommission Status : Normal
Configured Capacity: 254356226048 (236.89 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 115378343936 (107.45 GB)
DFS Remaining: 135124455424 (125.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Sat Oct 07 15:43:12 UTC 2023
Last Block Report: Sat Oct 07 15:21:42 UTC 2023
Num of Blocks: 0
但是在yarn和资源管理器中只有一个节点active
$ yarn node -list
2023-10-07 15:47:05 INFO DefaultNoHARMFailoverProxyProvider:64 - Connecting to ResourceManager at resourcemanager/172.25.0.6:8032
Total Nodes:1
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
c8ac8dd71a0d:41135 RUNNING c8ac8dd71a0d:8042 0
如何将其他节点添加到主动节点?
1条答案
按热度按时间4dbbbstv1#
Datanodes和Nodemanager是两个不同的东西。添加更多Nodemanager以更新YARN
请记住,您的系统可能最终耗尽内存试图运行所有这些容器,并没有真正的意义,因为它仍然是一个单点故障(你的主机)和竞争的同一组资源(CPU,硬盘和RAM),所以所有这些容器可能比在本地安装Hadoop并让它完全访问您的系统要慢,而不是强迫它在同一个硬盘驱动器内复制数据并从同一个主机进行内部(“假远程”)网络调用