我正在尝试使用flume socialagent检索facebook数据。我已经成功地使用twitter代理检索twitter数据。
但在facebook的情况下,我在hdfs中什么都没有。
我的终端卡在这个阶段。当我终止这个过程时,我得到了这个
我的hdfs facebook文件夹是空的
SocialAgent.sources = FacebookHttpSource Twitter
SocialAgent.channels = FBmemoryChannel MemChannel
SocialAgent.sinks = fbHDFS HDFS
# For each one of the sources, the type is defined
SocialAgent.sources.FacebookHttpSource.type = org.apache.flume.source.http.HTTPSource
SocialAgent.sources.FacebookHttpSource.port = 51400
SocialAgent.sources.FacebookHttpSource.interceptors = Ts
SocialAgent.sources.FacebookHttpSource.interceptors.Ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
# The channel can be defined as follows.
SocialAgent.sources.FacebookHttpSource.channels = FBmemoryChannel
# Each sink's type must be defined
# Specify the channel the sink should use
SocialAgent.sinks.fbHDFS.channel = FBmemoryChannel
SocialAgent.sinks.fbHDFS.type = hdfs
SocialAgent.sinks.fbHDFS.hdfs.path = hdfs://localhost:9000/user/flume/facebook/%Y/%m/%d/%H/
SocialAgent.sinks.fbHDFS.hdfs.fileType = DataStream
SocialAgent.sinks.fbHDFS.hdfs.writeFormat = Text
SocialAgent.sinks.fbHDFS.hdfs.batchSize = 1000
SocialAgent.sinks.fbHDFS.hdfs.rollSize = 0
SocialAgent.sinks.fbHDFS.hdfs.rollCount = 10000
# Each channel's type is defined.
SocialAgent.channels.FBmemoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
SocialAgent.channels.FBmemoryChannel.capacity = 10000
SocialAgent.channels.FBmemoryChannel.transactionCapacity = 100
是否有其他flume代理来获取facebook数据。
1条答案
按热度按时间vshtjzan1#
配置中的org.apache.flume.source.http.httpsource不会向facebook发送任何获取数据的请求。它只是监听一个端口(在您的例子中是51400)并等待任何请求(https://flume.apache.org/flumeuserguide.html#http-(来源)
据我所知,目前还没有任何flume客户。