我正在构建一个Web应用程序,它需要一些长时间运行的任务在AWS ECS上使用celery 作为分布式任务队列。我面临的问题是,在ECS上运行的celery 工作器并没有接收来自SQS的任务,即使它似乎连接到SQS。
以下是ECS任务的日志。
/usr/local/lib/python3.8/site-packages/celery/platforms.py:797: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
-------------- celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal v5.0.1 (singularity)
--- ***** -----
-- ******* ---- Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.2.5 2021-12-14 06:39:58
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: emptive_portal:0x7fbfda752310
- ** ---------- .> transport: sqs://XXXXXXXXXXXXXXXX:**@localhost//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> emptive-celery2.fifo exchange=sync(direct) key=emptive-celery.fifo
[tasks]
. import_export_celery.tasks.run_export_job
. import_export_celery.tasks.run_import_job
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] Connected to sqs://XXXXXXXXXXXXXXXXX:**@localhost//
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal ready.
字符串
需要注意的是,我已经在本地运行了部署到ECS的容器,与发送任务的django web服务器所在的同一台机器上运行了。那个celery 工人在接任务方面没有任何问题。
我也试过给予ecsTaskExecutionRole对SQS的完全权限,但这似乎没有影响任何东西。如果你能帮忙的话,我将不胜感激。
编辑:忘记在django显示我的celery 经纪人配置settings.py
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
# SQS CONFIG
CELERY_BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-2',
'polling_interval': 20,
}
CELERY_RESULT_BACKEND = None
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
型
1条答案
按热度按时间3htmauhk1#
所以我终于修好了。这个问题对我来说真的很愚蠢。:)我只需要在celery 配置中将BROKER_TRANSPORT_OPTIONS替换为CELERY_BROKER_TRANSPORT_OPTIONS。
新配置:
字符串