我试图发布一个非常简单的Scrapy蜘蛛作为一个EXE文件使用PyInstaller。
我已经搜索并阅读了我能找到的所有内容,但我仍然不知道哪里出了问题。我该如何解决这个问题?
如果我将yield改为return,它不会给予我错误,除了它只返回一个项目(这是正常的,因为它是return而不是yield),代码在我的IDE中工作得很好,没有任何错误(不使用PyInstaller EXE文件)。
注意:我使用的是PyInstaller开发版本。
运行EXE文件时出错:
2020-04-28 11:57:30 [scrapy.core.scraper] ERROR: Spider error processing <GET http://books.toscrape.com/> (referer: None)
Traceback (most recent call last):
File "lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
File "lib\site-packages\scrapy\core\downloader\middleware.py", line 42, in process_request
File "lib\site-packages\twisted\internet\defer.py", line 1362, in returnValue
twisted.internet.defer._DefGen_Return: <200 http://books.toscrape.com/>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "lib\site-packages\scrapy\utils\defer.py", line 55, in mustbe_deferred
File "lib\site-packages\scrapy\core\spidermw.py", line 60, in process_spider_input
File "lib\site-packages\scrapy\core\scraper.py", line 148, in call_spider
File "lib\site-packages\scrapy\utils\misc.py", line 202, in warn_on_generator_with_return_value
File "lib\site-packages\scrapy\utils\misc.py", line 187, in is_generator_with_return_value
File "inspect.py", line 973, in getsource
File "inspect.py", line 955, in getsourcelines
File "inspect.py", line 786, in findsource
OSError: could not get source code
文件 myBookSpider.py
import scrapy
from items import scrapyStandaloneTestItem
class bookSpider(scrapy.Spider):
name = "bookSpider"
custom_settings = {
"FEED_URI": "resultFile.csv",
"FEED_FORMAT": "csv",
"FEED_EXPORT_FIELDS": ["title", "price"]
}
def start_requests(self):
urls = [
"http://books.toscrape.com/",
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
# Getting an instance of our item class
item = scrapyStandaloneTestItem()
# Getting all the article's with product pod class
articles = response.css("article.product_pod")
# Looping through all the article elements we got earlier
for article in articles:
# Getting the needed values from the site and putting them in variables
title = article.css("a::attr(title)").extract()
price = article.css("p.price_color::text").extract()
# Setting the title / price variables in our items class equal to the variables that we just extracted data in to
item["title"] = title
item["price"] = price
yield item
文件 items.py
import scrapy
class scrapyStandaloneTestItem(scrapy.Item):
# Define the fields for your item here
title = scrapy.Field()
price = scrapy.Field()
文件 runSpider.py
# In this file we will run the spider(s)
from scrapy.crawler import CrawlerProcess
from myBookSpider import bookSpider
from scrapy.utils.project import get_project_settings
def runSpider():
# Running scraper
process = CrawlerProcess(get_project_settings())
process.crawl(bookSpider)
process.start()
if (__name__ == "__main__"):
runSpider()
1条答案
按热度按时间voase2hg1#
您所要做的就是将以下代码添加到您的spider中: