apache—使用flume向hdfs写入tweet对代理不起作用

9rnv2umw  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(355)

这是我的twitter.conf:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = <<API key>>
TwitterAgent.sources.Twitter.consumerSecret = <<API secret>>
TwitterAgent.sources.Twitter.accessToken = <<Access token>>
TwitterAgent.sources.Twitter.accessTokenSecret = <<Access token secret>>
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://Singh:9000/flume/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

这是我的flume.log:

20 Sep 2014 14:22:02,286 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
20 Sep 2014 14:22:02,297 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/home/vijay/BigData/flume-1.5.0.1/conf/twitter.conf
20 Sep 2014 14:22:02,307 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,308 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,308 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: HDFS Agent: TwitterAgent
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,309 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:HDFS
20 Sep 2014 14:22:02,322 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [TwitterAgent]
20 Sep 2014 14:22:02,322 WARN  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138)  - No configuration found for this host:agent
20 Sep 2014 14:22:02,330 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

你能帮我一下吗?

o4hqfura

o4hqfura1#

尝试此Flume配置:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = <<API key>>
TwitterAgent.sources.Twitter.consumerSecret = <<API secret>>
TwitterAgent.sources.Twitter.accessToken = <<Access token>>
TwitterAgent.sources.Twitter.accessTokenSecret = <<Access token secret>>

TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:54310/user/bigdata/tweets
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 10000

你需要加上http://files.cloudera.com/samples/flume-sources-1.0-snapshot.jar 将这个jar放入flume/lib文件夹

相关问题