apache flume目录配置失败

bjp0bcyl  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(463)

我使用以下代码将源目录中的文件写入hdfs。


# Initialize agent's source, channel and sink

agent.sources = test
agent.channels = memoryChannel
agent.sinks = flumeHDFS

# Setting the source to spool directory where the file exists

agent.sources.test.type = spooldir
agent.sources.test.spoolDir = /Data

# Setting the channel to memory

agent.channels.memoryChannel.type = memory

# Max number of events stored in the memory channel

agent.channels.memoryChannel.capacity = 10000

# agent.channels.memoryChannel.batchSize = 15000

agent.channels.memoryChannel.transactioncapacity = 1000000

# Setting the sink to HDFS

agent.sinks.flumeHDFS.type = hdfs
agent.sinks.flumeHDFS.hdfs.path = /user/team
agent.sinks.flumeHDFS.hdfs.fileType = DataStream

# Write format can be text or writable

agent.sinks.flumeHDFS.hdfs.writeFormat = Text

# use a single csv file at a time

agent.sinks.flumeHDFS.hdfs.maxOpenFiles = 1

# rollover file based on maximum size of 10 MB

agent.sinks.flumeHDFS.hdfs.rollCount=0
agent.sinks.flumeHDFS.hdfs.rollInterval=2000
agent.sinks.flumeHDFS.hdfs.rollSize = 0
agent.sinks.flumeHDFS.hdfs.batchSize =1000000

# never rollover based on the number of events

agent.sinks.flumeHDFS.hdfs.rollCount = 0

# rollover file based on max time of 1 min

# agent.sinks.flumeHDFS.hdfs.rollInterval = 0

# agent.sinks.flumeHDFS.hdfs.idleTimeout = 600

# Connect source and sink with channel

agent.sources.TwitterExampleDir.channels = memoryChannel
agent.sinks.flumeHDFS.channel = memoryChannel

但我得到以下错误
:配置组件失败!org.apache.flume.conf.configurationexception:配置组件失败!在org.apache.flume.conf.source.sourceconfiguration.configure(sourceconfigurati on。java:110)在org.apache.flume.conf.flumeconfiguration$agentconfiguration.validatesources(flumeconfiguration。java:566)在org.apache.flume.conf.flumeconfiguration$agentconfiguration.isvalid(flumecon配置)。java:345)在org.apache.flume.conf.flumeconfiguration$agentconfiguration.access$000(flume配置)。java:212)在org.apache.flume.conf.flumeconfiguration.validateconfiguration(flumeconfiguration。java:126)在org.apache.flume.conf.flumeconfiguration。java:108)在org.apache.flume.node.propertiesfileconfigurationprovider.getFlumeConfiguration(属性文件配置提供程序)。java:193)在org.apache.flume.node.abstractconfigurationprovider.getconfiguration(abstrac tconfigurationprovider)上。java:94)位于org.apache.flume.node.pollingpropertiesfileconfigurationprovider$filewatcherrunnable.run(pollingpropertiesfileconfigurationprovider)。java:140)在java.util.concurrent.executors$runnableadapter.call(executors。java:511)在java.util.concurrent.futuretask.runandreset(futuretask。java:308)在java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$301(scheduledthreadpoolexecutor。java:180)在java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(sch eduledthreadpoolexecutor。java:294)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:11 42)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:6 17)在java.lang.thread.run(线程。java:745)原因:org.apache.flume.conf.configurationexception:在org.apache.flume.conf.source.sourceconfiguration.configure(sourceconfigurati on)没有为测试设置通道。java:68) ... 还有15个
有人能帮我做些什么把我的数据从源目录/data放到hdfs目录/user/team吗?

0ejtzxu1

0ejtzxu11#

堆栈跟踪:

No channels set for test

您有一个指定的 sources 作为 test ```
agent.sources = test

但是在连接到 `channel` ```
agent.sources.TwitterExampleDir.channels = memoryChannel

所以你必须提到 test 而不是 TwitterExampleDir

相关问题