Scrapy PyInstaller OSError:无法获取源代码/ twisted.internet.defer._DefGen_Return

nfg76nw0  于 2023-03-30  发布在  其他
关注(0)|答案(1)|浏览(170)

我试图发布一个非常简单的Scrapy蜘蛛作为一个EXE文件使用PyInstaller。
我已经搜索并阅读了我能找到的所有内容,但我仍然不知道哪里出了问题。我该如何解决这个问题?
如果我将yield改为return,它不会给予我错误,除了它只返回一个项目(这是正常的,因为它是return而不是yield),代码在我的IDE中工作得很好,没有任何错误(不使用PyInstaller EXE文件)。
注意:我使用的是PyInstaller开发版本。
运行EXE文件时出错:

2020-04-28 11:57:30 [scrapy.core.scraper] ERROR: Spider error processing <GET http://books.toscrape.com/> (referer: None)
Traceback (most recent call last):
  File "lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
  File "lib\site-packages\scrapy\core\downloader\middleware.py", line 42, in process_request
  File "lib\site-packages\twisted\internet\defer.py", line 1362, in returnValue
twisted.internet.defer._DefGen_Return: <200 http://books.toscrape.com/>

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "lib\site-packages\scrapy\utils\defer.py", line 55, in mustbe_deferred
  File "lib\site-packages\scrapy\core\spidermw.py", line 60, in process_spider_input
  File "lib\site-packages\scrapy\core\scraper.py", line 148, in call_spider
  File "lib\site-packages\scrapy\utils\misc.py", line 202, in warn_on_generator_with_return_value
  File "lib\site-packages\scrapy\utils\misc.py", line 187, in is_generator_with_return_value
  File "inspect.py", line 973, in getsource
  File "inspect.py", line 955, in getsourcelines
  File "inspect.py", line 786, in findsource
OSError: could not get source code

文件 myBookSpider.py

import scrapy
from items import scrapyStandaloneTestItem

class bookSpider(scrapy.Spider):

    name = "bookSpider"
    custom_settings = {
        "FEED_URI": "resultFile.csv",
        "FEED_FORMAT": "csv",
        "FEED_EXPORT_FIELDS": ["title", "price"]
    }

    def start_requests(self):

        urls = [
            "http://books.toscrape.com/",
        ]

        for url in urls:

            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        # Getting an instance of our item class
        item = scrapyStandaloneTestItem()

        # Getting all the article's with product pod class
        articles = response.css("article.product_pod")

        # Looping through all the article elements we got earlier
        for article in articles:

            # Getting the needed values from the site and putting them in variables
            title = article.css("a::attr(title)").extract()
            price = article.css("p.price_color::text").extract()

            # Setting the title / price variables in our items class equal to the variables that we just extracted data in to
            item["title"] = title
            item["price"] = price
            yield item

文件 items.py

import scrapy

class scrapyStandaloneTestItem(scrapy.Item):

    # Define the fields for your item here
    title = scrapy.Field()
    price = scrapy.Field()

文件 runSpider.py

# In this file we will run the spider(s)
from scrapy.crawler import CrawlerProcess
from myBookSpider import bookSpider
from scrapy.utils.project import get_project_settings

def runSpider():

    # Running scraper
    process = CrawlerProcess(get_project_settings())
    process.crawl(bookSpider)
    process.start()

if (__name__ == "__main__"):

    runSpider()
voase2hg

voase2hg1#

您所要做的就是将以下代码添加到您的spider中:

import scrapy.utils.misc
import scrapy.core.scraper

def warn_on_generator_with_return_value_stub(spider, callable):
    pass

scrapy.utils.misc.warn_on_generator_with_return_value = warn_on_generator_with_return_value_stub
scrapy.core.scraper.warn_on_generator_with_return_value = warn_on_generator_with_return_value_stub

相关问题