使用flume从twitter提供数据时twitter4j类出错

uz75evzq  于 2021-06-03  发布在  Flume
关注(0)|答案(3)|浏览(506)

我使用的是github上传的cloudera flume-sources-1.0-snapshot提供的源代码。
我得到下面的错误。twitter4j中cloudera提供的jar的configuration.class中不存在该方法。
如何解决这个问题?

{    

    14/06/12 03:04:56 INFO node.PollingPropertiesFileConfigurationProvider:                                             Configuration provider starting
    14/06/12 03:04:56 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:conf/tagent
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Processing:HDFS
    14/06/12 03:04:56 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent]
    14/06/12 03:04:56 INFO node.AbstractConfigurationProvider: Creating channels
    14/06/12 03:04:56 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory
    14/06/12 03:04:57 INFO node.AbstractConfigurationProvider: Created channel MemChannel
    14/06/12 03:04:57 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource
    14/06/12 03:04:57 ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error
    java.lang.NoSuchMethodError:         twitter4j.conf.Configuration.getRequestHeaders()Ljava/util/Map;
at         twitter4j.StreamingReadTimeoutConfiguration.getRequestHeaders(TwitterStreamImpl.java:664)
at twitter4j.internal.http.HttpClientWrapper.<init>(HttpClientWrapper.java:47)
at twitter4j.TwitterStreamImpl.<init>(TwitterStreamImpl.java:54)
at twitter4j.TwitterStreamFactory.<clinit>(TwitterStreamFactory.java:40)
at com.cloudera.flume.source.TwitterSource.<init>(TwitterSource.java:64)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

   }
lokaqttq

lokaqttq1#

您必须替换放置在中的所有twitter4j-3.0.3库 flume/lib 文件夹 twitter4j-2.x.x 版本。原因是相同的类名在 twitter4j-3.0.3 以及 flume-sources-1.0-SNAPSHOT.jar 为了避免这种情况,您必须在twitter4j属性中进行更改,方法是将以下属性放入 twitter4j.properties 作为类路径:

streamBaseURL=https://stream.twitter.com/1.1/ `

或者可以从这里下载新的flume-source-1.0-snapshot.jar。
这个问题已经被twitter社区解决了,想要得到完整的解释,你可以查看这个讨论。

gg0vcinb

gg0vcinb2#

当你收到

twitter4j.conf.Configuration.getRequestHeaders() method error.

尝试以下步骤,它解决了我的问题。
从中删除以下3个jar文件 /usr/lib/flume-ng/lib/ directory .
twitter4j-core-3.0.3.jar
twitter4j-media-support-3.0.3.jar
twitter4j-stream-3.0.3.jar
将下面的jar文件添加到 /usr/lib/flume-ng/lib/ directory .
( twitter4j jar应该是最新版本。)
flume-sources-1.0-snapshot.jar
twitter4j-core-4.0.7.jar
twitter4j-async-4.0.7.jar
twitter4j-stream-4.0.7.jar

oprakyz7

oprakyz73#

检查您正在使用的twitter4jjars版本。
twitter4j.conf.configuration.getrequestheaders()方法不在最新版本中。
但它在twitter4j2.x中

相关问题