django 在ECS任务上运行celery 工人并使用SQS作为代理

2hh7jdfx  于 2023-08-08  发布在  Go
关注(0)|答案(1)|浏览(127)

我正在构建一个Web应用程序,它需要一些长时间运行的任务在AWS ECS上使用celery 作为分布式任务队列。我面临的问题是,在ECS上运行的celery 工作器并没有接收来自SQS的任务,即使它似乎连接到SQS。
以下是ECS任务的日志。

/usr/local/lib/python3.8/site-packages/celery/platforms.py:797: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
  warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
 
 -------------- celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal v5.0.1 (singularity)
--- ***** ----- 
-- ******* ---- Linux-4.14.252-195.483.amzn2.x86_64-x86_64-with-glibc2.2.5 2021-12-14 06:39:58
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         emptive_portal:0x7fbfda752310
- ** ---------- .> transport:   sqs://XXXXXXXXXXXXXXXX:**@localhost//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> emptive-celery2.fifo exchange=sync(direct) key=emptive-celery.fifo
                
[tasks]
 
  . import_export_celery.tasks.run_export_job
  . import_export_celery.tasks.run_import_job
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] Connected to sqs://XXXXXXXXXXXXXXXXX:**@localhost//
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
[Errno 2] No such file or directory: 'seq_tokens/emptive-staging_web_sequence_token.txt'
2021-12-14 06:39:58 [INFO] celery@ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal ready.

字符串
需要注意的是,我已经在本地运行了部署到ECS的容器,与发送任务的django web服务器所在的同一台机器上运行了。那个celery 工人在接任务方面没有任何问题。
我也试过给予ecsTaskExecutionRole对SQS的完全权限,但这似乎没有影响任何东西。如果你能帮忙的话,我将不胜感激。
编辑:忘记在django显示我的celery 经纪人配置settings.py

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    # SQS CONFIG
    CELERY_BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    CELERY_ACCEPT_CONTENT = ['application/json']
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_TASK_SERIALIZER = 'json'
    BROKER_TRANSPORT_OPTIONS = {
        'region': 'us-east-2',
        'polling_interval': 20,
    }
    CELERY_RESULT_BACKEND = None
    CELERY_ENABLE_REMOTE_CONTROL = False
    CELERY_SEND_EVENTS = False

3htmauhk

3htmauhk1#

所以我终于修好了。这个问题对我来说真的很愚蠢。:)我只需要在celery 配置中将BROKER_TRANSPORT_OPTIONS替换为CELERY_BROKER_TRANSPORT_OPTIONS。
新配置:

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    # SQS CONFIG
    CELERY_BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    CELERY_ACCEPT_CONTENT = ['application/json']
    CELERY_RESULT_SERIALIZER = 'json'
    CELERY_TASK_SERIALIZER = 'json'
    CELERY_BROKER_TRANSPORT_OPTIONS = {
        'region': 'us-east-2',
        'polling_interval': 20,
    }
    CELERY_RESULT_BACKEND = None
    CELERY_ENABLE_REMOTE_CONTROL = False
    CELERY_SEND_EVENTS = False

字符串

相关问题