Kafka流:Kafka流应用程序卡在重新平衡

xfb7svmp  于 2021-06-08  发布在  Kafka
关注(0)|答案(0)|浏览(243)

在所有kafka代理重新启动以升级offset.retention.minutes设置(将其增加到60天)之后,使用该设置的kafka streams应用程序被卡住,并且消费组显示重新平衡:

bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort 

Warning: Consumer group 'stream-processor' is rebalancing.

TOPIC                  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
customers              0          84263           84288           25              -               -               -
customers              1          71731           85068           13337           -               -               -
customers              10         71841           84801           12960           -               -               -
customers              11         84273           84336           63              -               -               -
customers              12         84189           84297           108             -               -               -
customers              13         83969           84051           82              -               -               -
customers              14         84693           84767           74              -               -               -
customers              15         84472           84556           84              -               -               -
customers              2          84175           84239           64              -               -               -
customers              3          71719           71719           0               -               -               -
customers              4          71446           84499           13053           -               -               -
customers              5          84291           84361           70              -               -               -
customers              6          71700           71700           0               -               -               -
customers              7          72003           85235           13232           -               -               -
customers              8          84521           84587           66              -               -               -
customers              9          71513           71513           0               -               -               -
customers-intermediate 0          102774          102792          18              -               -               -
customers-intermediate 1          102931          103028          97              -               -               -
customers-intermediate 10         102883          102965          82              -               -               -
customers-intermediate 11         102723          102861          138             -               -               -
customers-intermediate 12         102797          102931          134             -               -               -
customers-intermediate 13         102339          102460          121             -               -               -
customers-intermediate 14         103227          103321          94              -               -               -
customers-intermediate 15         103231          103366          135             -               -               -
customers-intermediate 2          102637          102723          86              -               -               -
customers-intermediate 3          84620           103297          18677           -               -               -
customers-intermediate 4          102596          102687          91              -               -               -
customers-intermediate 5          102980          103071          91              -               -               -
customers-intermediate 6          84385           103058          18673           -               -               -
customers-intermediate 7          103559          103652          93              -               -               -
customers-intermediate 8          103243          103312          69              -               -               -
customers-intermediate 9          84211           102772          18561           -               -               -
events                 15         11958555        15231834        3273279         -               -               -
events                 3          1393386         16534651        15141265        -               -               -
events                 4          1149540         15390069        14240529        -               -               -
visitors               15         2774874         2778873         3999            -               -               -
visitors               3          603242          603324          82              -               -               -
visitors               4          565266          565834          568

流应用程序也被重新启动了,之后我可以看到一些处理日志大约20个小时,然后停止处理。
已经这样两天了。但也值得一提的是,您在上面看到的所有主题都有16个分区,但有些分区显示了其中的三个分区(访客、事件)。不过,我可以描述的主题,他们有他们的分区分布正常,我可以发现没有什么奇怪的。
会发生什么事?在应用程序重新启动之后,我可以再次看到所有分区,以及从主题分区消费的应用程序。然而,许多(大多数)分区丢失了它们的偏移量。自从我换了衣服 offset.retention.minutes 设置,这不应该发生。

events                 0          -               14538261        -               stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
events                 1          49070           13276094        13227024        stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 10         -               15593746        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 11         -               15525487        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 12         -               21863908        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 13         -               15810925        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 14         -               13509742        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 15         11958555        15231834        3273279         stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
...

Kafka1.1.0,Kafka流1.0
更新仍在2.1.1上进行。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题