在并行线程中运行的kafka python使用者

mefy6pfw  于 2021-06-08  发布在  Kafka
关注(0)|答案(2)|浏览(395)

我对python和Kafka完全是个新手。我有一个脚本,应该启动三个kafka消费者,等待这些消费者的消息,然后做一些其他的事情。在这一点上,我甚至不知道我是否朝着正确的方向前进,所以任何帮助都将受到感谢。

class MainClass():
    def do_something_before(self):
        # something is done here

    def start_consumer(self):
        consumer1_thread = threading.Thread(target=self.cons1, args=())
        consumer2_thread = threading.Thread(target=self.cons2, args=())
        consumer1_thread.daemon = True
        consumer2_thread.daemon = True
        consumer1_thread.start()
        consumer2_thread.start()

    def cons1(self):
        consumer = KafkaConsumer(bootstrap_servers='localhost:9092',
                                 auto_offset_reset='earliest')
        consumer.subscribe(['my-topic'])
        for message in consumer:
            print(message.value)

    def cons2(self):
        consumer = KafkaConsumer(bootstrap_servers='localhost:9092',
                                 auto_offset_reset='earliest')
        consumer.subscribe(['my2-topic'])
        for message in consumer:
            print(message.value)

    def keep_working(self):
        # something is done here

if __name__ == 'main':
    g = MainClass()
    g.do_something_before()
    g.keep_working()
5us2dqdw

5us2dqdw1#

我已经添加了python-kafka示例和2个使用者(基本上是两个python进程),您可以在github链接中找到它https://github.com/shubhamgorde/kafka-python-app.
不能发布整个python文件,它有点大。

from multiprocessing import Process

def consumeData(topic):
    try:
         consumer = KafkaConsumer(topic, value_deserializer=lambda v: 
           binascii.unhexlify(v).decode('utf-8'))
    except:
         print("Error!!")

    for msg in consumer:
        msg=ast.literal_eval(msg.value)
        if(msg[2] == 'C'):
            performCreditOperation(msg)
        elif (msg[2] == 'D'):
              performDebitOperation(msg)

t1 = Process(target=consumeData, args=('Credit_transac',))
t2 = Process(target=consumeData, args=('Debit_transac',))
t1.start()
t2.start()
lrl1mhuk

lrl1mhuk2#

这是我的实现。希望你觉得有用。

class ConsumerThread:
    def __init__(self, config, topics):
        self.config = config
        self.topics = topics

    def readData(self):
        consumer = Consumer(self.config)
        consumer.subscribe(self.topics)
        self.run(consumer)

    def process_msg(self, msg):
        print('Received message.')
        print('Key: {}, Val: {}'.format(msg.key(), msg.value()))
        print('Partition: {}, Offset: {}'.format(msg.partition(), msg.offset()))

    def run(self, consumer):
        try:
            while True:
                msg = consumer.poll(0.1)
                if not msg:
                    continue
                if msg.error():
                    if msg.error().code() == KafkaError._PARTITION_EOF:
                        # End of partition event
                        print('End of partition reached {0}/{1}'
                            .format(msg.topic(), msg.partition()))
                    else:
                        raise KafkaException(msg.error())
                else:
                    self.process_msg(msg)

        except KeyboardInterrupt:
            print("Detected Keyboard Interrupt. Cancelling.")
            pass

        finally:
            consumer.close()

相关问题