我在RabbitMQ中有一个任务队列,其中有多个生产者(12个)和一个消费者,用于Web应用程序中的繁重任务。当我运行消费者时,它开始将一些消息出队,然后崩溃并出现以下错误:
Traceback (most recent call last):
File "jobs.py", line 42, in <module> jobs[job](config)
File "/home/ec2-user/project/queue.py", line 100, in init_queue
channel.start_consuming()
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1822, in start_consuming
self.connection.process_data_events(time_limit=None)
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 749, in process_data_events
self._flush_output(common_terminator)
File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 477, in _flush_output
result.reason_text)
pika.exceptions.ConnectionClosed: (-1, "error(104, 'Connection reset by peer')")
生产者代码是:
message = {'image_url': image_url, 'image_name': image_name, 'notes': notes}
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks_queue')
channel.basic_publish(exchange='', routing_key=queue_name, body=json.dumps(message))
connection.close()
唯一的消费者代码(一个是冲突):
def callback(self, ch, method, properties, body):
"""Callback when receive a message."""
message = json.loads(body)
try:
image = _get_image(message['image_url'])
except:
sys.stderr.write('Error getting image in note %s' % note['id'])
# Crop image with PIL. Not so expensive
box_path = _crop(image, message['image_name'], box)
# API call. Long time function
result = long_api_call(box_path)
if result is None:
sys.stderr.write('Error in note %s' % note['id'])
return
# update the db
db.update_record(result)
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks_queue')
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback_obj.callback, queue='tasks_queue', no_ack=True)
channel.start_consuming()
正如你所看到的,有3个昂贵的消息功能。一个裁剪任务、一个API调用和一个数据库更新。没有API调用,que consumer运行顺畅。
Thanks in advance
2条答案
按热度按时间yvfmudvl1#
你的RabbitMQ日志显示了一条我认为我们可能会看到的消息:
你的
long_api_call
阻塞了Pika的I/O循环。Pika是一个非常轻量级的库,不会在后台启动线程,所以你必须以这样一种方式编写代码,以免阻塞Pika的I/O循环超过心跳间隔。RabbitMQ认为您的客户端已经死亡或没有响应,并强制关闭连接。请参阅my answer here,它链接到this example code,展示了如何在单独的线程中正确执行长时间运行的任务。您仍然可以使用
no_ack=True
,只需跳过ack_message
调用即可。pgvzfuti2#
从RabbitMQ 3.5.5开始,代理的默认心跳超时从580秒减少到60秒。
请参见pika: Ensuring well-behaved connection with heartbeat and blocked-connection timeouts。
最简单的修复方法是增加心跳超时: