未在hdfs接收器中创建flumedata文件

kqlmhetl  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(384)

我正在尝试使用kafka作为源,flume作为汇来接收实时数据。汇类型是hdfs。我的producer工作正常,我可以看到正在生成的数据,我的代理运行正常(运行命令时没有错误),但是文件没有在指定的目录中生成。
启动flume代理的命令:

/usr/hdp/2.5.0.0-1245/flume/bin/flume-ng agent -c /usr/hdp/2.5.0.0-1245/flume/conf -f /usr/hdp/2.5.0.0-1245/flume/conf/flume-hdfs.conf -n tier1

以及我的flume-hdfs.conf文件:

tier1.sources  = source1
 tier1.channels = channel1
 tier1.sinks = sink1

 tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
 tier1.sources.source1.zookeeperConnect = localhost:2181
 tier1.sources.source1.topic = data_1
 tier1.sources.source1.channels = channel1

 tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
 tier1.channels.channel1.brokerList = localhost:6667
 tier1.channels.channel1.zookeeperConnect = localhost:2181
 tier1.channels.channel1.capacity = 10000
 tier1.channels.channel1.transactionCapacity = 1000

 tier1.sinks.sink1.type = hdfs
 tier1.sinks.sink1.hdfs.path = /user/user_name/FLUME_LOGS/
 tier1.sinks.sink1.hdfs.rollInterval = 5
 tier1.sinks.sink1.hdfs.rollSize = 0
 tier1.sinks.sink1.hdfs.rollCount = 0
 tier1.sinks.sink1.hdfs.fileType = DataStream
 tier1.sinks.sink1.channel = channel1

我无法找出处决的问题所在。
请建议如何克服这个问题。

flvtvl50

flvtvl501#

按以下方式设置hdfs接收器的路径:

tier1.sinks.sink1.hdfs.path = "VALUE of fs.default.name, located in core-site.xml"/user/user_name/FLUME_LOGS/

例如

tier1.sinks.sink1.hdfs.path = hdfs://localhost:54310/user/user_name/FLUME_LOGS/

相关问题