我没有收到twitter的推文

col17t5w 于 2021-06-02 发布在 Hadoop

关注(0)|答案(2)|浏览(297)

我尝试通过flume在hdfs中传输tweets，我没有从twitter获得tweets。这个过程在下面提到的这一点上停止！！除了这一步什么都没有。

INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully.
INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started

我的flume.conf文件：

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type =       org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey =**************
TwitterAgent.sources.Twitter.consumerSecret =*********
TwitterAgent.sources.Twitter.accessToken =************
TwitterAgent.sources.Twitter.accessTokenSecret =*********

TwitterAgent.sources.Twitter.keywords = big data,cloud,hadoop,data analysis,hive,pig,hbase,flume,datawarehouse
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/Big
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

hadoop hdfs flume bigdata twitter

来源：https://stackoverflow.com/questions/30681210/i-am-not-getting-the-tweets-from-twitter

2条答案

按热度按时间

yacmzcpb1#

尝试此代码。。

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxx

TwitterAgent.sources.Twitter.keywords = big data,cloud,hadoop,data analysis,hive,pig,hbase,flume,datawarehouse

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/Big
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.filePrefix = twitter_India-
TwitterAgent.sinks.HDFS.hdfs.round = true
TwitterAgent.sinks.HDFS.hdfs.roundValue = 2
TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

赞(0）回复(0）举报 2021-06-03

2skhul332#

在 flume.conf 文件

TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:**9000**/Big

将您自己的活动localhost instade设置为9000。
去http://localhost：50070/您可以看到活动的localhost。就像我的情况一样

TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:54310/user/flume/tweets

赞(0）回复(0）举报 2021-06-03

我来回答

我没有收到twitter的推文

2条答案

相关问题

热门标签

最新问答