flume不使用socialagent获取facebook数据

u3r8eeie  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(479)

我正在尝试使用flume socialagent检索facebook数据。我已经成功地使用twitter代理检索twitter数据。
但在facebook的情况下,我在hdfs中什么都没有。

我的终端卡在这个阶段。当我终止这个过程时,我得到了这个

我的hdfs facebook文件夹是空的

SocialAgent.sources = FacebookHttpSource Twitter
SocialAgent.channels = FBmemoryChannel MemChannel
SocialAgent.sinks = fbHDFS HDFS

# For each one of the sources, the type is defined

SocialAgent.sources.FacebookHttpSource.type = org.apache.flume.source.http.HTTPSource
SocialAgent.sources.FacebookHttpSource.port = 51400
SocialAgent.sources.FacebookHttpSource.interceptors = Ts
SocialAgent.sources.FacebookHttpSource.interceptors.Ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

# The channel can be defined as follows.

SocialAgent.sources.FacebookHttpSource.channels = FBmemoryChannel

# Each sink's type must be defined

# Specify the channel the sink should use

SocialAgent.sinks.fbHDFS.channel = FBmemoryChannel
SocialAgent.sinks.fbHDFS.type = hdfs
SocialAgent.sinks.fbHDFS.hdfs.path = hdfs://localhost:9000/user/flume/facebook/%Y/%m/%d/%H/
SocialAgent.sinks.fbHDFS.hdfs.fileType = DataStream
SocialAgent.sinks.fbHDFS.hdfs.writeFormat = Text
SocialAgent.sinks.fbHDFS.hdfs.batchSize = 1000
SocialAgent.sinks.fbHDFS.hdfs.rollSize = 0
SocialAgent.sinks.fbHDFS.hdfs.rollCount = 10000

# Each channel's type is defined.

SocialAgent.channels.FBmemoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

SocialAgent.channels.FBmemoryChannel.capacity = 10000
SocialAgent.channels.FBmemoryChannel.transactionCapacity = 100

是否有其他flume代理来获取facebook数据。

vshtjzan

vshtjzan1#

配置中的org.apache.flume.source.http.httpsource不会向facebook发送任何获取数据的请求。它只是监听一个端口(在您的例子中是51400)并等待任何请求(https://flume.apache.org/flumeuserguide.html#http-(来源)
据我所知,目前还没有任何flume客户。

相关问题