scrapy 如何使用Python从eBay上抓取/下载所有产品图片？

我只能从eBay网站刮一个全分辨率图像的网址;我无法捕捉所有其他图像的网址。我正在寻找一个脚本，刮或下载所有的图像。
我想要高分辨率的照片，而不是缩略图，下载。

from lxml import html  
import requests
from bs4 import BeautifulSoup
import pandas as pd
main_url= 'https://www.ebay.com/'
headers= {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36'
}
url= 'https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=laptop&_sacat=0&LH_TitleDesc=0&rt=nc&_odkw=toaster&_osacat=0&LH_PrefLoc=3&LH_All=1&_ipg=240'
r= requests.get(url, headers=headers)
print(r)
soup= BeautifulSoup(r.content, 'html.parser')

product_list= soup.find_all('div', class_= 's-item__image')

products_site = []

for item in product_list:
    for link in item.find_all('a', href= True):
        products_site.append(link['href'])
products_site = list(dict.fromkeys(products_site))
products_site = list(filter(None, products_site))        
products_site = [x for x in products_site if x.startswith('https://www.ebay.com/itm/')][:2]
print(len('product_site'))

item_list=[]
for link in products_site:
    r = requests.get(link, headers=headers)
    print(r)
    soup= BeautifulSoup(r.content, 'html.parser')
    Title= soup.select_one('h1', class_='x-item-title__mainTitle').get_text(strip=True)
    Image_URL= [x['src'] for x in soup.findAll('img', {'id': 'icImg'})]
    Product= {
        "Title": Title,        
        "Image_URL": Image_URL
    
        }

字符串

图片的URL在eBay上保持不变。
要获得高分辨率的产品的所有图像，您可以轻松更改这些不同缩略图的尺寸并获得HQ图像。
举例来说，

https://i.ebayimg.com/images/g/pxcAAOSwis1hwW4V/s-l64.jpg

字符串
.jpg之前的尾部s-l64表示分辨率为64 p，您可以将其更改为s-l100/s-l300或s-l500以增加分辨率，它支持的最高分辨率为-s-l2000。
因此，您只需将缩略图的s-l64替换为s-l2000即可获得HQ图像。
使用这个技巧，你不需要点击图像放大，并获得总部的图像。
完整的工作代码-

import requests
from bs4 import BeautifulSoup

main_url = 'https://www.ebay.com/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36'
}
url = 'https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=laptop&_sacat=0&LH_TitleDesc=0&rt=nc&_odkw=toaster&_osacat=0&LH_PrefLoc=3&LH_All=1&_ipg=240'
r = requests.get(url, headers=headers)
print(r)
soup = BeautifulSoup(r.content, 'html.parser')

product_list = soup.find_all('div', class_='s-item__image')

products_site = []

for item in product_list:
    for link in item.find_all('a', href=True):
        products_site.append(link['href'])
products_site = list(dict.fromkeys(products_site))
products_site = list(filter(None, products_site))
products_site = [x for x in products_site if x.startswith('https://www.ebay.com/itm/')][:2]
print(len('product_site'))

item_list = []
for link in products_site:
    # print(link)
    r = requests.get(link, headers=headers)
    print(r)
    soup = BeautifulSoup(r.content, 'html.parser')
    Title = soup.select_one('h1', class_='x-item-title__mainTitle').get_text(strip=True)

    # example page - https://www.ebay.com/itm/125058259597?epid=4051542538&hash=item1d1e0d9a8d:g:pxcAAOSwis1hwW4V

    image_urls = [i.get('src').replace('s-l64', 's-l2000')
                  for i in soup.select('ul#vertical-align-items-viewport > li img')]
    if len(image_urls) == 0:
        # example page with no extra images
        # https://www.ebay.com/itm/125287169558?epid=19053326726&hash=item1d2bb27e16:g:sRAAAOSwKV9ia3Ia

        image_urls = set([x['src'] for x in soup.findAll('img', {'id': 'icImg'})])  # remove duplicate images
    product = {
        "Title": Title,
        "Image_URL": image_urls

    }
    print(product)

型
输出-

<Response [200]>
12
<Response [200]>
{'Title': 'Lenovo Legion 5 Pro 16 165Hz QHD IPS G-Sync Ryzen 7 16GB RAM 1TB SSD RTX 3070', 'Image_URL': ['https://i.ebayimg.com/images/g/pxcAAOSwis1hwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/UWEAAOSwLslhwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/sOIAAOSwANNhwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/SOIAAOSwwORhwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/g7kAAOSwhzNhwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/HjsAAOSw6pxhvXmX/s-l2000.jpg', 'https://i.ebayimg.com/images/g/OSQAAOSwAvVhwW4V/s-l2000.jpg', 'https://i.ebayimg.com/images/g/pHAAAOSwjnJhwW4V/s-l2000.jpg', '//p.ebaystatic.com/aw/pics/cmp/icn/iconImgNA_96x96.gif', '//p.ebaystatic.com/aw/pics/cmp/icn/iconImgNA_96x96.gif']}
<Response [200]>
{'Title': '\ufeff\ufeffLenovo IdeaPad Gaming 3 15.6" 120Hz i5-11300H 8GB RAM 512GB SSD GTX 1650', 'Image_URL': {'https://i.ebayimg.com/images/g/sRAAAOSwKV9ia3Ia/s-l500.jpg'}}

型

scrapy 如何使用Python从eBay上抓取/下载所有产品图片？

1条答案

相关问题

热门标签

最新问答