public <T> InputDStream<T> queueStream(scala.collection.mutable.Queue<RDD<T>> queue,
boolean oneAtATime,
scala.reflect.ClassTag<T> evidence$15)
Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.
NOTE: Arbitrary RDDs can be added to queueStream, there is no way to recover data of those RDDs, so queueStream doesn't support checkpointing.
1条答案
按热度按时间j13ufse21#
试试QueueStreamAPI。
rdd队列作为一个流,推送到队列中的每个rdd都将被视为数据流中的一批数据,并像流一样进行处理。