如何使用Scrapy抓取所有URL？

brc7rcf0  于 2022-11-09  发布在  其他
关注(0)|答案(1)|浏览(157)

我尝试通过以下方式获取搜索结果文章的URL：

selector = response.xpath("//*[contains(@class, 'bw-news-list')]/a/@href").extract()
selector = response.xpath("//*[contains(@class, 'bw-search-results')]/a/@href").extract()
selector = response.css('ul.bw-news-list a::attr(href)')

但我一个也没找到。
这是网站URL：https://www.businesswire.com/portal/site/home/search/?searchType=all&searchTerm=amputee&searchPage=1

scrapy

1条答案

按热度按时间
kfgdxczn

kfgdxczn1#

因为网页是以AJAX请求的形式从外部源（API）动态加载的，所以您得到的结果集为空。因此您必须改用API url。

以工作溶液为例：

import scrapy
class TestSpider(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        headers= {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
            'x-requested-with': 'XMLHttpRequest'
        }

        url='https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken'
        yield scrapy.Request(
            url=url,
            headers=headers,
            callback= self.parse,
            method="GET")

    def parse(self, response):
        for link in response.xpath('(//*[@class="bw-news-list"])[1]/li/h3/a/@href'):
            yield {
                'URL': link.get()
            }

输出：

{'URL': 'http://www.businesswire.com/news/home/20220810005639/en/The-Amputee-Coalition%E2%80%99s-National-Conference-Welcomes-over-650-Attendees-in-the-Limb-Loss-and-Limb-Difference-Community'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220809005759/en/Napoli-Shkolnik-PLLC-files-50-million-lawsuit-against-Suffolk-County'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220727006087/en/Dialyze-Direct-Partners-with-CommuniCare-Health-to-Provide-On-Site-Dialysis'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220602005023/en/Indian-Motorcycle-Partners-With-Veterans-Charity-Ride-for-8th-Annual-Motorcycle-Therapy-Program'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220421005541/en/GreenRoom-Advisory-Lee-Puts-His-Only-Foot-Forward-%E2%80%93-Again%21'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220315005628/en/CORRECTING-and-REPLACING-NextGen-Healthcare-Teams-Up-with-Three-time-Paralympic-Medalist-Amy-Purdy-to-Champion-Whole-Person-Care'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220314005652/en/Global-3D-Printing-Markets-2022---2027-by-Printer-Type-Materials-Software-Applications-Services-and-Solutions-in-Industry-Verticals---ResearchAndMarkets.com'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220130005017/zh-CN/'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220130005018/zh-HK/'}
2022-08-25 14:36:35 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.businesswire.com/portal/site/home/template.BINARYPORTLET/search/resource.process/?javax.portlet.tpst=92055fbcbec7e639f1f554100d908a0c&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchTerm=amputee&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_resultsPage=1&javax.portlet.rst_92055fbcbec7e639f1f554100d908a0c_searchType=news&javax.portlet.rid_92055fbcbec7e639f1f554100d908a0c=searchPaging&javax.portlet.rcl_92055fbcbec7e639f1f554100d908a0c=cacheLevelPage&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken>
{'URL': 'http://www.businesswire.com/news/home/20220202005449/en/2021-Pain-Management-Devices-Pipeline-Product-Landscape-Report---ResearchAndMarkets.com'}
赞(0）分享  回复(0）举报  2022-11-09
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com