在所有kafka代理重新启动以升级offset.retention.minutes设置(将其增加到60天)之后,使用该设置的kafka streams应用程序被卡住,并且消费组显示重新平衡:
bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort
Warning: Consumer group 'stream-processor' is rebalancing.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customers 0 84263 84288 25 - - -
customers 1 71731 85068 13337 - - -
customers 10 71841 84801 12960 - - -
customers 11 84273 84336 63 - - -
customers 12 84189 84297 108 - - -
customers 13 83969 84051 82 - - -
customers 14 84693 84767 74 - - -
customers 15 84472 84556 84 - - -
customers 2 84175 84239 64 - - -
customers 3 71719 71719 0 - - -
customers 4 71446 84499 13053 - - -
customers 5 84291 84361 70 - - -
customers 6 71700 71700 0 - - -
customers 7 72003 85235 13232 - - -
customers 8 84521 84587 66 - - -
customers 9 71513 71513 0 - - -
customers-intermediate 0 102774 102792 18 - - -
customers-intermediate 1 102931 103028 97 - - -
customers-intermediate 10 102883 102965 82 - - -
customers-intermediate 11 102723 102861 138 - - -
customers-intermediate 12 102797 102931 134 - - -
customers-intermediate 13 102339 102460 121 - - -
customers-intermediate 14 103227 103321 94 - - -
customers-intermediate 15 103231 103366 135 - - -
customers-intermediate 2 102637 102723 86 - - -
customers-intermediate 3 84620 103297 18677 - - -
customers-intermediate 4 102596 102687 91 - - -
customers-intermediate 5 102980 103071 91 - - -
customers-intermediate 6 84385 103058 18673 - - -
customers-intermediate 7 103559 103652 93 - - -
customers-intermediate 8 103243 103312 69 - - -
customers-intermediate 9 84211 102772 18561 - - -
events 15 11958555 15231834 3273279 - - -
events 3 1393386 16534651 15141265 - - -
events 4 1149540 15390069 14240529 - - -
visitors 15 2774874 2778873 3999 - - -
visitors 3 603242 603324 82 - - -
visitors 4 565266 565834 568
流应用程序也被重新启动了,之后我可以看到一些处理日志大约20个小时,然后停止处理。
已经这样两天了。但也值得一提的是,您在上面看到的所有主题都有16个分区,但有些分区显示了其中的三个分区(访客、事件)。不过,我可以描述的主题,他们有他们的分区分布正常,我可以发现没有什么奇怪的。
会发生什么事?在应用程序重新启动之后,我可以再次看到所有分区,以及从主题分区消费的应用程序。然而,许多(大多数)分区丢失了它们的偏移量。自从我换了衣服 offset.retention.minutes
设置,这不应该发生。
events 0 - 14538261 - stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
events 1 49070 13276094 13227024 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 10 - 15593746 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 11 - 15525487 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 12 - 21863908 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 13 - 15810925 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 14 - 13509742 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 15 11958555 15231834 3273279 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
...
Kafka1.1.0,Kafka流1.0
更新仍在2.1.1上进行。
暂无答案!
目前还没有任何答案,快来回答吧!