Flex框中项目的Scrapy xpath或CSS

brccelvz 于 2023-06-23 发布在其他

关注(0)|答案(1)|浏览(143)

完成Scrapy noob和成功地通过教程后，我试图在网站上刮一个页面，我正在收集数据以进行进一步分析，但我试图使用的xpath不断刮取什么也没有返回。我唯一能告诉你的不同之处是文本/URL在页面上的Flex框中。我已经尝试了每一个迭代，似乎工作在元素搜索栏和什么都没有。我是否缺少一个函数来允许访问flexbox中的这些行？
我正在抓取的页面的URL：https://partsmasterusa.com/product-category/crown/page/2/
我在粗糙的shell中尝试过的XPath示例没有提供任何结果：

response.xpath('//div[@id="main"]/div/div/div/div/main/div[@class="archive-products"]//div[@class="product-content"]/a/text()').get()

response.xpath('//div[@id="main"]/div/div/div/div/main/div[@class="archive-products"]//div[@class="product-content"]/a/text()').extract_first()

response.xpath('//li[contains(@class, "product-col")]//a[@href]/text()').extract_first()
etc etc

一个XPath响应的示例，它返回了我所期望的结果：

response.xpath('//div//a/span/text()').extract_first()

TIA

scrapy

来源：https://stackoverflow.com/questions/76443472/scrapy-xpath-or-css-for-items-inside-a-flex-box

1条答案

按热度按时间

bnlyeluc1#

这个网站有点棘手。这是一个wordpress网站，所以它不是完全动态的，但是在每个页面上呈现的产品是 AJAX 调用加载的。因此，当您尝试提取时，您可能试图从网站中提取的内容并不存在。
你可以做的是复制页面触发的POST请求，以便通过浏览器开发工具的网络选项卡将所有产品加载到页面上。
在调查之后，您会发现它向https://partsmasterusa.com/product-category/crown/page/{page_number_here}/?count=36发出请求。通过复制这个url和请求头以及在请求体中发送的字段和值，您可以通过这种方式获得所有产品及其信息。
例如：

import scrapy
import json

class PartSpider(scrapy.Spider):
    name = "partsmaster"

    def start_requests(self):
        url = "https://partsmasterusa.com/product-category/crown/page/{}/?count=36"
        body = {"portoajax": True, "load_posts_only": True}
        headers = {"X-Requested-With": "XMLHttpRequest"}
        for i in range(1,542):
            yield scrapy.Request(url.format(i), method="POST", body=json.dumps(body), headers=headers)

    def parse(self, response):
        for lnk in response.xpath("//a[@class='product-loop-title']"):
            yield {"title": lnk.xpath('./h3/text()').get()
                   "url": lnk.xpath('./@href').get()}

部分输出：

2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Insulator (093603)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'KEY RING  (107763)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Key Switch  (146289)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Key Switch Assembly (146286)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Kit Label EEC (126271-(1))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL (69395)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Contactor Component Map(869419-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Control Component Map(869421-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL KIT  (126270)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Power Component Map(869420-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Pump Motor Map(869422-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Drive Module Map(69444)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Drive Module Map(869444-(1))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Motor Map(869423-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-CONTACTOR CONTROL MAP (69419)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-CONTROL COMPONENT MAP (69421)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-POWER COMPONENT MAP (69420)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-PUMP MOTOR MAP (69422)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-TRACTION MOTOR MAP (69423)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Load Wheel  (077086-201)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Load Wheel Assembly Includes Bearings  (093656-201)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Locknut (060043–008)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LOCKWASHER  (060005-003)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Lockwasher Without Quick Coast  (060005-045)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LT SILVER GRADE ANTI-SEIZE  (065005-003)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Manual Coast Selector Warning  (069100)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MODULE AC4820 FIN RR5200 (142885-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MODULE VCM RR5200 AC SERVICE  (129325-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Module Warning Label(69376)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Momentum RR Decal (069372-001.)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MOTOR – HYDRAULIC  (121659)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Motor Nameplate (021062-008)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Mount Wheel  (084009)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Mount Wheel  (115388)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Moving Contact  (114435)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Moving Vehicle Warning  (069004)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '#4 REG SPLIT LCW  (060005-049)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '1/4 INT LCW  (060005-022)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '350 BLUE HSNG  (078723-006)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'A.C.-TRACTION DRIVE MODULE  (130056)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'AC RR52 MOTOR 36V TRACTION (21067)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'AC RR52 MOTOR 36V TRACTION (21187)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 1 Display  (146688)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 1 MODULE NEW- (140131)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 2 MODULE NEW- (142517-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 3 Label(69375)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 4 MODULE NEW- (141779-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 5 MODULE NEW- (143911-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS MODULE 2 (121611-00S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ArmInner Primary (12.2924-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ASM HEATER RR5000 24V  (129132-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Axle  (080191)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Axle  (116804)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130528)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130529)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130530)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130531)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130532)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Bearing (.065081-045)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING – BALL SENSOR  (130692)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING SLEEVE  (130701)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING- (065081-043)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BLOCK – TERMINAL (21053)'}

赞(0）回复(0）举报 2023-06-23

我来回答

Flex框中项目的Scrapy xpath或CSS

1条答案

相关问题

热门标签

最新问答