Flume:KafkaFlume不能得到所有的推特

myzjeezk  于 2021-06-04  发布在  Flume
关注(0)|答案(0)|浏览(259)

我正在尝试从twitterapi收集tweet,并通过flume将它们写到kafka。我用KafkaFlume。问题是kafka接收器没有收集flume收集的所有tweet。我运行zookeeper和kafka服务器,创建了主题twitter,并与消费者一起收听了主题。例如,对于flume收集的1000条tweet,经过1分钟的处理后,kafka只显示100条。这是flume conf文件:

TwitterAgent.sources = Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks = kafka

TwitterAgent.sources.Twitter.type =   org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = 
TwitterAgent.sources.Twitter.consumerSecret = 
TwitterAgent.sources.Twitter.accessToken = 
TwitterAgent.sources.Twitter.accessTokenSecret = 
TwitterAgent.sources.Twitter.channels = MemChannel

TwitterAgent.sinks.kafka.channel = MemChannel
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10020
TwitterAgent.channels.MemChannel.transactionCapacity = 1300

TwitterAgent.sinks.kafka.type = org.apache.flume.sink.kafka.KafkaSink
TwitterAgent.sinks.kafka.topic = twitter
TwitterAgent.sinks.kafka.brokerList = localhost:9092
TwitterAgent.sinks.kafka.batchsize = 100
TwitterAgent.sinks.kafka.request.required.acks = -1

谢谢你的帮助

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题