我是新来的 Flume . 我有 Flume 以及 Hadoop 安装在一台服务器上，而日志在另一台服务器上可用。
通过 Flume ，我正在看日志。这是我的配置文件。
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.type = syslogtcp
agent1.sources.avro-source1.bind = 10.209.4.224
agent1.sources.avro-source1.port = 5140
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.hdfs-sink1.type = hdfs
agent1.sinks.hdfs-sink1.hdfs.path = hdfs://delvmplldsst02:54310/flume/events
agent1.sinks.hdfs-sink1.hdfs.fileType = DataStream
agent1.sinks.hdfs-sink1.hdfs.writeFormat = Text
agent1.sinks.hdfs-sink1.hdfs.batchSize = 20
agent1.sinks.hdfs-sink1.hdfs.rollSize = 0
agent1.sinks.hdfs-sink1.hdfs.rollCount = 0
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = hdfs-sink1
# chain the different components together
agent1.sinks.hdfs-sink1.channel = ch1
agent1.sources.avro-source1.channels = ch1我不确定在这个场景中使用什么样的源类型。我要开始了 Flume agent 在另一个服务器中如下所示：
bin/flume-ng agent --conf-file conf/flume.conf -f /var/log/wtmp -Dflume.root.logger=DEBUG,console -n agent1这是上面命令的日志
14/06/25 00:37:17 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
14/06/25 00:37:17 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:conf/flume.conf
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink1 Agent: agent1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Processing:hdfs-sink1
14/06/25 00:37:17 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
14/06/25 00:37:17 INFO node.AbstractConfigurationProvider: Creating channels
14/06/25 00:37:17 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory
14/06/25 00:37:17 INFO node.AbstractConfigurationProvider: Created channel ch1
14/06/25 00:37:17 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type syslogtcp
14/06/25 00:37:17 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink1, type: hdfs
14/06/25 00:37:17 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
14/06/25 00:37:17 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, hdfs-sink1]
14/06/25 00:37:17 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:avro-source1,state:IDLE} }} sinkRunners:{hdfs-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5954864a counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
14/06/25 00:37:17 INFO node.Application: Starting Channel ch1
14/06/25 00:37:17 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean.
14/06/25 00:37:17 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started
14/06/25 00:37:17 INFO node.Application: Starting Sink hdfs-sink1
14/06/25 00:37:17 INFO node.Application: Starting Source avro-source1
14/06/25 00:37:17 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs-sink1: Successfully registered new MBean.
14/06/25 00:37:17 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink1 started
14/06/25 00:37:17 INFO source.SyslogTcpSource: Syslog TCP Source starting...在这里，ptoccess被卡住了，根本没有进一步的进展。我不知道哪里会出错
有人能帮我做同样的事吗
我没有在有日志文件的服务器上安装flume。我要在那里也装Flume吗？？
flume版本使用-1.5.0 hadoop版本安装-1.0.4
提前谢谢
2条答案
按热度按时间ergxz8rk1#
问题可能来自系统日志。您的flume似乎启动良好，它看起来空闲的原因是它没有从syslog接收任何事件。
确保syslog守护进程正在向port=5140发送事件，对于agent1.sources.avro-source1.bind，可以通过将ip替换为0.0.0.0绑定到任何源（如果计划从多个服务器侦听）
你可以去登记
/etc/rsyslog.conf. @霍斯特纳meofflume:flumesourceport in 你的案子应该是*.* @10.209.4.224:5140（假设此ip是您的flume主机的ip）yvt65v4c2#
您需要将另一台服务器配置为将其syslog输出转发到日志服务器。这种配置完全取决于您运行的syslog守护进程。
日志输出使我觉得它启动正确。