无法从“scrapy_playwright.page”导入名称“PageCoroutine”

ubby3x7f 于 2022-11-09 发布在其他

关注(0)|答案(1)|浏览(216)

我尝试使用scrapy和剧作家刮动态网页，我安装了scrapy和剧作家，但是，当我试图运行我的蜘蛛，我得到这个错误。
ImportError: cannot import name 'PageCoroutine' from 'scrapy_playwright.page' (C:\Ali\DataCamp\Web Scraping in Python\Scrapy\venv\lib\site-packages\scrapy_playwright\page.py)
这是我的代码（这是一个测试代码）：

import scrapy
from scrapy_playwright.page import PageCoroutine

class PwspiderSpider(scrapy.Spider):
    name = 'pwspider'

    def start_requests(self):
        yield scrapy.Request("https://shoppable-campaign-demo.netlify.app/#/", meta=dict(playwright=True, playwright_include_page=True, playwright_page_coroutine=[PageCoroutine('wait_for_selector', 'div#productListing')]))

    async def parse(self, response):
        yield {'text': response.text}

我甚至在设置文件中添加了DOWNLOAD_HANDLERS和TWISTED_REACTOR。

scrapy

来源：https://stackoverflow.com/questions/74328303/cannot-import-name-pagecoroutine-from-scrapy-playwright-page

1条答案

按热度按时间

qq24tv8q1#

PageCoroutine已过时。请改用playwright_page_methods。

工作代码示例：

import scrapy
from scrapy_playwright.page import PageMethod

class TestSpider(scrapy.Spider):
    name = "test"
    def start_requests(self):
        yield scrapy.Request(

            url="https://shoppable-campaign-demo.netlify.app/#/",
            callback=self.parse,
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", '.card-body'),
                ],
            },
        )

    def parse(self, response):

        products = response.xpath('//*[@class="card-body"]')
        for product in products:
            yield {
            'title':product.xpath('.//*[@class="card-title"]/text()').get()

            }

输出：

{'title': 'Oxford Loafers'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Ankle-length Slack'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'White Baseball Cap'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Triangle Bikini Top'}
2022-11-05 20:40:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://shoppable-campaign-demo.netlify.app/#/>
{'title': 'Short Blazer'}
2022-11-05 20:40:40 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-05 20:40:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 39851,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 41.370211,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 11, 5, 14, 40, 40, 261151),
 'item_scraped_count': 5,

赞(0）回复(0）举报 2022-11-09

我来回答

无法从“scrapy_playwright.page”导入名称“PageCoroutine”

1条答案

相关问题

热门标签

最新问答