无法pickle本地对象“run\u spider.< locals>.f”

guicsvcw  于 2021-09-29  发布在  Java
关注(0)|答案(0)|浏览(182)

我一直在尝试在spyder上运行下面的代码,这是我在这个问题上发现的:

import scrapy
import scrapy.crawler as crawler
from multiprocessing import Process, Queue
from twisted.internet import reactor

# your spider

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['http://quotes.toscrape.com/tag/humor/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            print(quote.css('span.text::text').extract_first())

# the wrapper to make it run more times

def run_spider(spider):
    def f(q):
        try:
            runner = crawler.CrawlerRunner()
            deferred = runner.crawl(spider)
            deferred.addBoth(lambda _: reactor.stop())
            reactor.run()
            q.put(None)
        except Exception as e:
            q.put(e)

    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    result = q.get()
    p.join()

    if result is not None:
        raise result

print('first run:')
run_spider(QuotesSpider)

print('\nsecond run:')
run_spider(QuotesSpider)

但是,当我运行它时,会出现以下错误:

AttributeError: Can't pickle local object 'run_spider.<locals>.f'

我看到了一个答案

Had small issue regarding 'AttributeError: Can't pickle local object 'run_spider.<locals>.f', but moving function called f outside resolved my issue, and I could run the code –

我试图通过将函数f放置在 run_spider 函数,甚至在不同的文件中。但仍然不起作用。
任何帮助都将不胜感激。非常感谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题