flume内存通道在启动时已满

o4hqfura  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(726)

我的flume内存通道有问题。我运行了一个flume代理,它淹没了内存通道,日志开始溢出“通道已满,现在无法写入数据。源将在250毫秒后重试“
到现在为止,一直都还不错。我停止代理,编辑flume.conf以增加容量,然后重试。问题是,在启动时,flume已经溢出了相同的消息:

`16/05/14 00:21:48 INFO node.Application: Starting new configuration:
 { sourceRunners:{s1=EventDrivenSourceRunner: { source:Spool  
  Directory source s1: { spoolDir: /home/vagrant/logs } }} 
  sinkRunners:{kafka-avro-sink2=SinkRunner: { 
  policy:org.apache.flume.sink.DefaultSinkProcessor@63203b59 
  counterGroup:{ name:null counters:{} } }, kafka-avro-
   sink1=SinkRunner: { 
   policy:org.apache.flume.sink.DefaultSinkProcessor@591882e6 
  counterGroup:{ name:null counters:{} } }} channels:
  {mem1=org.apache.flume.channel.MemoryChannel{name: mem1}} }
  16/05/14 00:21:48 INFO node.Application: Starting Channel mem1
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:       
  Monitored counter group for type: CHANNEL, name: mem1: Successfully  
  registered new MBean.
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Component type: CHANNEL, name: mem1 started
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro- 
  sink2
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-sink1
  16/05/14 00:21:48 INFO node.Application: Starting Source s1
  16/05/14 00:21:48 INFO source.SpoolDirectorySource:  
  SpoolDirectorySource source starting with directory:  
  /home/vagrant/logs
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Monitored counter group for type: SOURCE, name: s1: Successfully  
  registered new MBean.
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
   Component type: SOURCE, name: s1 started
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after 
  250 milliseconds
  16/05/14 00:21:49 INFO avro.ReliableSpoolingFileEventReader: Last  
  read was never committed - resetting mark position.
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after  
  500 milliseconds`

所以当我启动代理时,频道已经满了。如何手动重置或清除?
为了一个类似的问题,我找了好几个小时都没有找到。一个稍微恼人的问题是https://flume.apache.org/ 几乎所有与flume相关的内容都被google编入索引。因此,搜索“Flume频道总是满的”,例如,将始终给出https://flume.apache.org/ 作为第一个结果。

erhoui1w

erhoui1w1#

似乎无法为频道分配足够的内存。当事件的传入速率高于内存通道的可用内存时。
尝试使用下面的例子:xmx是一个示例选项,您可以根据并行运行的代理数逐渐增加内存分配的最佳值。
例如:

flume-ng agent -n $agentnumber -c ../../config/conf/ -f ../../config/conf/youragentconf.conf -Xmx3g

此外,您还需要通过引用flume指南中的这个内存通道来检查代理配置文件中的其他配置参数。

相关问题