我正在尝试在flume配置中创建一个hivesink，当我运行flume ng时，会遇到一些依赖性问题，如下所示。你能告诉我怎样才能克服这种依赖吗？看起来有一些运行时依赖关系。我已经正确安装了hive，并进行了必要的环境变量设置，以将hive\u指向家中。感谢您的帮助。谢谢。
2016-01-15 14:41:37,757 (conf-file-poller-0)
[INFO -org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]
Creating instance of sink: hiveSink, type: hive
2016-01-15 14:41:37,763 (conf-file-poller-0)
[ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunn able.run(PollingPropertiesFileConfigurationProvider.java:145)]
Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter
at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:223)
at org.apache.flume.sink.hive.HiveSink.configure(HiveSink.java:203)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
2条答案
按热度按时间8ehkhllq1#
我正在学习CDH5.7发行版的flume。我也遇到了同样的问题，flume-env.sh脚本似乎不适合我，所以我使用--classpath命令参数来引用所有与hive和hivehcatalog相关的库。遵循反复试验的方法添加多个lib文件夹引用。没有找到太多关于这个论点的文献。
例如。
flume-ng agent --conf /home/cloudera/flume/ --conf-file /home/cloudera/flume/netcat_memchannel_hivesink.conf --name agent1 --classpath "/usr/lib/hive-hcatalog/share/hcatalog/*":"/usr/lib/hive/lib/*"xqkwcwgp2#
您应该通过flume-env.sh设置hive\u home和hcat\u home环境变量，或者将它们提供给用户配置文件。