twitter接收器未在spark 1.6.0中运行

x8goxv8g 于 2021-07-13 发布在 Java

关注(0)|答案(1)|浏览(359)

出于某种原因，如果我包括一个twitter接收器并启动流式处理上下文，我会遇到以下异常不确定为什么有人能让我知道是否有人已经遇到了这个问题或我做错了什么？

java.lang.ArithmeticException: / by zero 
        at org.apache.spark.streaming.Duration.isMultipleOf(Duration.scala:59) 
        at org.apache.spark.streaming.dstream.DStream.isTimeValid(DStream.scala:324) 
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344) 
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344) 
        at scala.Option.orElse(Option.scala:257) 
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:341) 
        at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:47) 
        at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:115) 
        at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:114) 
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) 
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) 
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) 
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) 
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) 
        at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:114) 
        at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:248) 
        at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:246) 
        at scala.util.Try$.apply(Try.scala:161) 
        at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:246) 
        at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:181) 
        at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:87) 
        at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:86) 
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 
2016-03-28 10:08:00,112 ERROR [DefaultQuartzScheduler_Worker-8] org.quartz.core.JobRunShell 
Job sample_TwitterListener.SocialMedia_sample threw an unhandled Exception:

Java apache-spark spark-streaming twitter4j

来源：https://stackoverflow.com/questions/36257012/twitter-receiver-not-running-in-spark-1-6-0

1条答案

按热度按时间

wwwo4jvm1#

终于找到了问题所在，在代码中我初始化了流上的两个窗口，如下所示

JavaReceiverInputDStream<Status> inputStream = TwitterUtils.createStream(jssc, getAuth());
JavaDStream<String> batchedInput = rawInput.window(new Duration(windowInterval), new Duration(slideInterval));
processStreamData(batchedInput);

private void processStreamData(JavaDStream<String> _input) {
    JavaDStream<String> input = _input.window(new Duration(windowInterval), new Duration(slideInterval));
}

因此，第一个加窗的流用来获得正确的滑动间隔，但是不知怎么的，第二个窗口获得了0毫秒的滑动间隔，这是由零异常引起的。在删除第二个窗口操作之后，我能够接收twitter数据。

赞(0）回复(0）举报 2021-07-13

我来回答

twitter接收器未在spark 1.6.0中运行

1条答案

相关问题

热门标签

最新问答