我有一个问题,与flume频道它无法初始化日志文件在通道。我正试图猫的文件,并加载到本地目录使用flume。
下面是配置文件信息和日志文件信息
conf文件
#### Define a source, a channel, and a sink
agent.sources = src1
agent.channels = chan1
agent.sinks = sink1
#### Set the source type to Spooling Directory and set the directory
#### location to /home/flume/ingestion/
agent.sources.src1.type = exec
agent.sources.src1.command= cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat
#### Configure the channel as simple in-memory queue
agent.channels.chan1.type = file
agent.channels.chan1.checkpointDir = /home/xxxxx/xxxxxxx/flume/checkpoint
agent.channels.chan1.dataDirs= /home/xxxxxxx/xxxxx/flume/data
#### Define the HDFS sink and set its path to your target HDFS directory
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/xxxxx/xxxxxx/xxxxx/flume/files
agent.sinks.sink1.sink.rollInterval = 0
agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0
#### Connect source and sink
agent.sources.src1.channels = chan1
agent.sinks.sink1.channel = chan1
Flume日志文件
27 Apr 2017 10:49:40,503 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
27 Apr 2017 10:49:40,521 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/lib/gphd/apache-flume-1.4.0-bin/conf/flume-agent1.conf
27 Apr 2017 10:49:40,530 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 10:49:40,531 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 10:49:40,531 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: sink1 Agent: agent
27 Apr 2017 10:49:40,531 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 10:49:40,532 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 10:49:40,674 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [agent]
27 Apr 2017 10:49:40,675 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
27 Apr 2017 10:49:40,687 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel chan1 type file
27 Apr 2017 10:49:40,691 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel chan1
27 Apr 2017 10:49:40,692 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source src1, type exec
27 Apr 2017 10:49:40,701 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: sink1, type: file_roll
27 Apr 2017 10:49:40,706 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel chan1 connected to [src1, sink1]
27 Apr 2017 10:49:40,716 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{src1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:src1,state:IDLE} }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@27d03700 counterGroup:{ name:null counters:{} } }} channels:{chan1=FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }} }
27 Apr 2017 10:49:40,730 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel chan1
27 Apr 2017 10:49:40,733 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:282) - Starting FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }...
27 Apr 2017 10:49:40,747 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.<init>:322) - Encryption is not enabled
27 Apr 2017 10:49:40,748 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:369) - Replay started
27 Apr 2017 10:49:40,750 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:381) - Found NextFileID 0, from []
27 Apr 2017 10:49:40,782 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFile.<init>:91) - Preallocated /root/.flume/file-channel/checkpoint/checkpoint to 8008232 for capacity 1000000
27 Apr 2017 10:49:40,785 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53) - Starting up with /root/.flume/file-channel/checkpoint/checkpoint and /root/.flume/file-channel/checkpoint/checkpoint.meta
27 Apr 2017 10:49:40,813 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:476) - Failed to initialize Log on [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
at org.apache.flume.channel.file.Log.replay(Log.java:412)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 10:49:40,816 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:313) - Failed to start the file channel [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
at org.apache.flume.channel.file.Log.replay(Log.java:412)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 10:49:40,817 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink sink1
27 Apr 2017 10:49:40,818 INFO [lifecycleSupervisor-1-1] (org.apache.flume.sink.RollingFileSink.start:104) - Starting org.apache.flume.sink.RollingFileSink{name:sink1, channel:chan1}...
27 Apr 2017 10:49:40,818 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source src1
27 Apr 2017 10:49:40,819 INFO [lifecycleSupervisor-1-0] (org.apache.flume.source.ExecSource.start:163) - Exec source starting with command:cat /wadpfs4/filehub/DTCC/inbound/DTC.A001.dat
27 Apr 2017 10:49:40,820 INFO [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SINK, name: sink1, registered successfully.
你能帮我一下吗。
暂无答案!
目前还没有任何答案,快来回答吧!