使用BashOperator运行

• https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html

with DAG(<args>) as dag:
     scrapy_task = BashOperator(
          task_id='scrapy',
          bash_command='scrapy crawl spider_1')

• 如果您使用的是virtualenv，则可以使用VirtualEnvOperator
• 或者使用现有环境，您可以使用source activate venv && scrapy crawl spider_1

使用Python运算符运行

• 来自Scrapy文档：https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.crawl('spider_1')
process.start() # the script will block here until the crawling is finished