scrapy 提取网站，但它们提供错误的输出

kadbb459 于 2022-11-09 发布在其他

关注(0)|答案(1)|浏览(98)

我尝试提取表，他们会给予我输出，但他们会错了，这是页面链接https://hoopshype.com/salaries/players/

from scrapy.http import Request
import scrapy
class PushpaSpider(scrapy.Spider):
    name = 'pushpa'
    page_number = 1
    start_urls = ['https://hoopshype.com/salaries/players/']
    custom_settings = {
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }

    def parse(self, response):
       rows = response.xpath("//table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']//thead//tr")
       keys = rows.xpath(".//td/text()").getall()
       keys = [i.strip() for i in keys]

       keys = [i for i in keys if i]

       columns=response.xpath("//table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']//tbody//tr")
       for column in columns:
           players=column.xpath('td//text()').getall()
           players = ''.join(players).strip()
           details = dict(zip(keys, players))
           yield details

scrapy

来源：https://stackoverflow.com/questions/71385541/extract-website-but-they-provide-wrong-output

1条答案

按热度按时间

cpjpxq1n1#

试试看：

from scrapy.http import Request
import scrapy
class PushpaSpider(scrapy.Spider):
    name = 'pushpa'
    page_number = 1
    start_urls = ['https://hoopshype.com/salaries/players/']
    custom_settings = {
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }

    def parse(self, response):
       rows = response.xpath("//table/thead/tr")
       keys = rows.xpath(".//td/text()").getall()
       keys = [i.strip() for i in keys]

       keys = [i for i in keys if I]

       columns=response.xpath("//table/tbody/tr")
       for column in columns:
           player_name = [column.xpath('td[@class="name"]/a/text()').get().strip()]
           detail = column.xpath('td/@data-value').getall()
           details = dict(zip(keys, player_name+detail))
           yield details

data-value也保存了数值，所以我们也可以使用它。因为我在提取text()时遇到了问题。
最后，我认为您不需要指定表类名（table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']），因为页面只有一个表。

赞(0）回复(0）举报 2022-11-09

我来回答

scrapy 提取网站，但它们提供错误的输出

1条答案

相关问题

热门标签

最新问答