合流kafka python库consumer.poll(超时)未按预期工作

qnyhuwrf  于 2021-06-04  发布在  Kafka
关注(0)|答案(0)|浏览(455)

当我设置 msg = consumer.poll(timeout=10.0) 消费者等待10秒后返回 None 如预期,但当我把这个改成 msg = consumer.poll(timeout=3600.0) 这个消费者刚刚回来 None 而不是像预期的那样等待3600秒。我错过什么了吗?这是完整的代码,如果需要的话。

running = True
conf = {'bootstrap.servers': bootstrap_servers,
        'group.id': 'foo',
        'auto.offset.reset': 'earliest',
        'enable.auto.commit': False,
        'on_commit': commit_completed}
consumer = Consumer(conf)

def msg_process(msg):
    print(f"key: {msg.key().decode('utf-8')}, value: {msg.value().decode('utf-8')}")

def basic_consume_loop(consumer, topics):
    try:
        consumer.subscribe(topics)

        msg_count = 0
        while running:
            msg = consumer.poll(timeout=3600.0)
            if msg is None:
                print(f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}: no new message")
                continue

            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    # End of partition event
                    print('%% %s [%d] reached end at offset %d\n' %
                          (msg.topic(), msg.partition(), msg.offset()))
                elif msg.error():
                    raise KafkaException(msg.error())
            else:
                # consumer.commit(async=False)
                msg_process(msg)
                msg_count += 1
                if msg_count % MIN_COMMIT_COUNT == 0:
                    consumer.commit(async=True)
    finally:
        # Close down consumer to commit final offsets.
        consumer.close()

def shutdown():
    running = False

basic_consume_loop(consumer, [topic_user])

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题