kafka流消费者组并发:仅从一个分区消费?

tuwxkamq  于 2021-06-06  发布在  Kafka
关注(0)|答案(0)|浏览(197)

我有一个
3个docker容器上的1个kafka集群,在ecs上运行,使用efs作为存储服务(为简单起见)。
1个kafka streams应用程序集群,位于3个容器上。
有一个源主题“事件”,有16个分区,复制2。一个papi拓扑处理器“流处理器”产生输出到其他一些主题,并使用3个状态存储。
我可以通过kafka管理器看到数据被消耗,输出被生成到这些其他输出主题。很明显它起作用了(虽然很慢)。
但是通过bin/kafka-consumer-groups.sh查看消费者补偿,我可以看到随着时间的推移,只有一个分区同时被消费。在命令的分离、连续运行中,仅减少一个偏移量。
首次执行:

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          671945          6046546         5374601         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

第二次执行:只有分区8提前了其偏移量。在1、5或15分钟后,该分区是唯一消耗的分区。

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          686685          6046546         5359861         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

查看日志,只有一个示例同时打印日志。i、 e.如果一个在工作,其他两个就不工作。
有什么问题吗?
Kafka和Kafka流版本1.1。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题