无法在Scrapy中抓取网页的描述

jckbn6z7  于 2022-11-09  发布在  其他
关注(0)|答案(3)|浏览(141)

我想使用Scrapy抓取数据,这里是链接
https://bbdealz.com/product/1000pcs-jigsaw-puzzle-7550cm-with-storage-bag-wooden-paper-puzzles-educational-toys-for-children-bedroom-decoration-stickers/

我使用此代码获取Description

'description': response.css('#tab-description p::text').extract(),

但回应是

description': ['    ', 'None  ', '  ', 'Unisex  ', '  ', '12-15 Years  ', '  ', 'Grownups  ', '  ', 'Paper  ', '  ', 'Landscape  ', '  ', 'SMW783   ']

它会忽略<strong><br>标记。
我需要这样的输出

<p>    <strong>Brand Name: </strong>None  <br>  <strong>Gender: </strong>Unisex  <br>  <strong>Age Range: </strong>12-15 Years  <br>  <strong>Age Range: </strong>Grownups  <br>  <strong>Material: </strong>Paper  <br>  <strong>Style: </strong>Landscape  <br>  <strong>Model Number: </strong>SMW783   </p>
7tofc5zh

7tofc5zh1#

您可以尝试使用xpath:
我测试了这个,它似乎工作:

for element in response.xpath("//div[@id='tab-description']/p"):
    values = element.xpath("./text()").getall()
    labels = element.xpath("./strong/text()").getall()
    values = [i for i in values if i.strip()]
    data = {labels[i]:values[i] for i in range(len(labels))}
    print(data)

输出:{'Brand Name: ': 'None ', 'Gender: ': 'Unisex ', 'Age Range: ': 'Grownups ', 'Material: ': 'Paper ', 'Style: ': 'Landscape ', 'Model Number: ': 'SMW783 '}

这只是html:

a = response.xpath("//div[@id='tab-description']/p").get()
print(a)

输出功率

<p>    <strong>Brand Name: </strong>None  <br>  <strong>Gender: </strong>Unisex  <br>  <strong>Age Range: </strong>12-15 Years  <br>  <strong>Age Range: </strong>Grownups  <br>  <strong>Material: </strong>Paper  <br>  <strong>S
tyle: </strong>Landscape  <br>  <strong>Model Number: </strong>SMW783   </p>
bxjv4tth

bxjv4tth2#

当XPath中的/text()或CSS中的::text不能产生所需的结果时,我使用另一个库。
以安装它。

pip3 install html2text

这些例子

from html2text import HTML2Text
h = HTML2Text()
h.ignore_links = True
h.ignore_images = True
h.ignore_emphasis = True

# Inside the scrapy project

'description': h.handle(response.css('#tab-description p').get()).strip()

yield ....
fxnxkyjh

fxnxkyjh3#

我试过这个,它工作

import scrapy
from bs4 import BeautifulSoup

class Google(scrapy.Spider):
    name = 'google'
    start_urls = ['https://bbdealz.com/product/funny-sports-game-2m-3m-4m-5m-6m-diameter-outdoor-rainbow-umbrella-parachute-toy-jump-sack-ballute-play-game-mat-toy-kids-gift/',]
    def parse(self, response):
        soup = BeautifulSoup(response.body, 'html.parser')
        yield {
        'title': response.css('h1::text').get(),
        'description': soup.select_one('#tab-description').select_one('p'),
        }

此致

相关问题