我有一个flink集群,并用它来运行批处理作业。问题是每次作业结束后,taskmanager的内存使用仍然繁忙。
集群运行在docker swarm中,有两台250gb内存的机器。
我的flink配置如下:
by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.
jobmanager.rpc.address: jobmanager17
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The heap size for the JobManager JVM
jobmanager.heap.mb: 204800
# The heap size for the TaskManager JVM
taskmanager.heap.mb: 204800
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
taskmanager.numberOfTaskSlots: 32
# Specify whether TaskManager memory should be allocated when starting up (true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to set this value to `false`
# as the default state backends currently do not use the managed memory.
taskmanager.memory.preallocate: false
# The parallelism used for programs that did not specify and other parallelism.
parallelism.default: 1
集群工作得很好,但它在一个作业上花费内存,最后使用交换和磁盘结束。
我希望这是一个配置问题,但如果你需要更多的细节码头,请告诉我
暂无答案!
目前还没有任何答案,快来回答吧!