如何在从mongodb获取相同的数据后,加速kafka的速度?

fcg9iug3  于 2021-06-06  发布在  Kafka
关注(0)|答案(0)|浏览(149)

我正在从mongodb获取数据并将其放入kafka。357 responses/second是完成获取和发布的速率。
如何改进从mongodb获取的功能:

from kafka import KafkaProducer
from kafka.errors import KafkaError
import json
import pymongo
from pymongo import MongoClient
import sys

try:
  client = MongoClient('my_uri')
  db = client["xxx-dev"]
except Exception as e:
    print e
producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
producer = KafkaProducer(retries=5)
id = 1
for response in db.Response.find():
    try:        
        future = producer.send('collect-production-response', bytes(response))
    except Exception as e:
        print e
    id  += 1
    if(id >= 100000):
        print "Done 100k"
        producer.flush()
        sys.exit()

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题