我是新来的ApacheFlume。
我试图了解如何获取json(作为http源代码),解析它并根据内容将其存储到hdfs上的动态路径。
例如:
如果json是:
[{
"field1" : "value1",
"field2" : "value2"
}]
那么hdfs路径将是:
/某个默认根路径/value1/value2/某个值名称文件
有没有这样的Flume配置,使我能够做到这一点?
以下是我当前的配置(通过http接受json,并根据时间戳将其存储在路径中):
# flume.conf: http source, hdfs sink
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 9000
# a1.sources.r1.handler = org.apache.flume.http.JSONHandler
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /user/uri/events/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
谢谢!
1条答案
按热度按时间idv4meu81#
解决方案在hdfsFlume的Flume文档中:
以下是修改后的配置:
还有 curl :