使用flume向hdfs写入tweet不起作用

62o28rlo  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(355)

我在vmware中使用了cloudera cdh5 quickstart虚拟机,所有服务都是通过cloudera manager安装的。
我创建了一个/user/flume/tweets和一个flume用户和组。我重新启动了所有的服务,但是,不管我等多久,没有tweet会被写入hdfs。/user/flume/tweets/目录仍然为空!
为什么?
这是my flume.conf:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey =**
TwitterAgent.sources.Twitter.consumerSecret =**
TwitterAgent.sources.Twitter.accessToken =**
TwitterAgent.sources.Twitter.accessTokenSecret =***
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera,        data science, data scientiest, business intelligence, mapreduce, data warehouse, data     warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path =     hdfs://localhost.localdomain:804/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

这是我在Flume日志上看到的:

[cloudera@localhost ~]$ tail -f /var/log/flume-ng/flume.log
27 May 2014 21:40:28,536 INFO  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -     Processing:HDFS
27 May 2014 21:40:28,536 INFO  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -     Processing:HDFS
27 May 2014 21:40:28,536 INFO  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -     Processing:HDFS
27 May 2014 21:40:28,537 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -     Processing:HDFS
27 May 2014 21:40:28,537 INFO  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  -     Processing:HDFS
27 May 2014 21:40:28,562 WARN  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:319)  - Agent     configuration for 'agent' does not contain any channels. Marking it as invalid.
27 May 2014 21:40:28,564 WARN  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:127)  - Agent configuration     invalid for agent 'agent'. It will be removed.
27 May 2014 21:40:28,564 INFO  [conf-file-poller-0]     (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation     flume configuration contains configuration for agents: [TwitterAgent]
27 May 2014 21:40:28,564 WARN  [conf-file-poller-0]     (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138)  - No     configuration found for this host:agent
27 May 2014 21:40:28,592 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{     sourceRunners:{} sinkRunners:{} channels:{} }

我该怎么修?
提前谢谢。

wmomyfyw

wmomyfyw1#

您是否使用cloudera manager设置了flume配置?请点击此链接http://javet.org/?p=279 用于在cdh5中实现twitter firehose。

相关问题