spark streaming中创建多个avroFlume的问题

vnzz0bqm  于 2021-06-04  发布在  Flume
关注(0)|答案(0)|浏览(175)

我需要连接多个FlumeFlume与Spark流这是我的Flume文件:

agent1.sinks.sink1a.type = avro
agent1.sinks.sink1a.hostname = localhost
agent1.sinks.sink1a.port = 9091

agent1.sinks.sink1b.type = avro
agent1.sinks.sink1b.hostname = localhost
agent1.sinks.sink1b.port = 9092

但只有9091端口正在连接9092无法连接
以下是创建多个Flume流的spark代码:

val sparkConf = new SparkConf().setAppName("WordCount")
val ssc = new StreamingContext(sparkConf, Seconds(20))

val rawLines = FlumeUtils.createStream(ssc,"localhost", 9091)
val rawLines1 = FlumeUtils.createStream(ssc,"localhost", 9092)

val lines = rawLines.map{record => {
                (new String(record.event.getBody().array()))}}

val lines1 = rawLines1.map{record1 => {
                (new String(record1.event.getBody().array()))}}

val lines_combined = lines.union(lines1)

val words = lines_combined.flatMap(_.split(" "))

我做错什么了?如果能帮上忙就好了。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题