FlumehdfsFlume的辊设置

kiayqfof  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(424)

下面是我在flume.conf中的设置:

agent.sources = srcpv
agent.channels = chlpv
agent.sinks = hdfsSink
agent.sources.srcpv.type = exec
agent.sources.srcpv.command = tail -F /var/log/collector/web/pv.log
agent.sources.srcpv.channels = chlpv
agent.channels.chlpv.type = memory
agent.channels.chlpv.capacity = 1000000
agent.channels.chlpv.transactionCapacity = 100
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = chlpv
agent.sinks.hdfsSink.hdfs.path = hdfs://hadoop01:8020/flume/web/pv/
agent.sinks.hdfsSink.hdfs.filePrefix = pv-
agent.sinks.hdfsSink.hdfs.rollSize = 1024
agent.sinks.hdfsSink.hdfs.rollInterval= 30
agent.sinks.hdfsSink.hdfs.rollCount = 10

我希望文件可以按一定的大小或间隔滚动,但滚动设置(rollsize,rollinterval,rollcount)不能生效,没有生成hdfs文件。几分钟后我发现这个错误: [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.hdfs.HDFSEventSink - process failed java.lang.OutOfMemoryError: GC overhead limit exceeded 有人能帮忙指出合适的hdfs接收器设置吗?

czfnxgou

czfnxgou1#

运行flume时,似乎java内存不足。
您可以尝试在flume-env.sh文件中添加以下行: export JAVA_OPTS="-Xms100m -Xmx2g -Dcom.sun.management.jmxremote" 根据系统配置增加值xmx。
希望这有帮助:)

相关问题