scrapy AttributeError:'CrawlerRunner'对象没有属性'spiders'

pw136qt2  于 2023-11-19  发布在  其他
关注(0)|答案(1)|浏览(121)

我面临着一个小问题;我很久以前工作过一个棘手的项目,我想重新启动它。我有不同的蜘蛛,我把它们重新组合在一个主脚本中,启动它们。有趣的是,它当时正在工作,但现在我有一个属性错误(AttributeError:'CrawlerRunner'对象没有属性'spiders')。下面是我的代码:

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
from twisted.internet import reactor
from LesInfosDuJour.pipelines import LesinfosdujourPipeline

configure_logging()
settings = get_project_settings()
runner = CrawlerRunner(settings)

for spider_name in runner.spiders.list():
    runner.crawl(spider_name)

d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run()
obj = LesinfosdujourPipeline()
obj.empty_procedure()
obj.duplicate_procedure()

字符串
我用python启动它(python news-scrap.py)
setting.py

SPIDER_MODULES = ['LesInfosDuJour.spiders']
NEWSPIDER_MODULE = 'LesInfosDuJour.spiders'


项目目录:Scrap/LesInfosDuJour/spiders

ljsrvy3e

ljsrvy3e1#

经过一番研究,scrappy在最新的更新中删除了CrawlerRunner.spider方法,并将其替换为CrawlerRunner.spider_loader
以下是我做的改动:

configure_logging()
settings = get_project_settings()
runner = CrawlerRunner(settings)
spiders_list = list(runner.spider_loader.list())

for spider_name in spiders_list:
    runner.crawl(spider_name)

d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run()

字符串

相关问题