json 抓取评论的URL地址

因此，我需要从该网站上的产品的URL中提取评论，更具体地说，是用户名、日期、文本和分数。然而，我有一些问题，因为我不断得到一个错误：无法检索第% 1页的评论。错误：“连接断开：InvalidChunkLength（get length b''，0 bytes read）";“InvalidChunkLength（得到长度b ''，读取0字节）";我试着增加一个时间延迟，但它仍然不起作用。我如何修改它？

import json
import requests
from bs4 import BeautifulSoup

url = "https://www.emag.ro/covor-antiderapant-negru-poliester-80-x-300-cm-c027-80x300/pd/DBY5YJMBM/?ref=sponsored_products_fill_a_b_5_3&provider=rec&recid=rec_73_c449bb3e50b63cc8f6da4a42a31af359f6cbfb3c547bc5748cb6d45501a29685_1684315709&scenario_ID=73&aid=034a897a-956c-11ed-9004-0ab644dfda7c&oid=89847310"

review_url = "https://www.emag.ro/review/get-review-listing-page?id={product_id}&page={page}"
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0'}

product_id = url.split("/pd/")[1].split("/")[0]

reviews = []

page = 1
while True:
    r_url = review_url.format(product_id=product_id, page=page)
    try:
        response = requests.get(r_url, headers=headers)
        response.raise_for_status()  
        data = response.json()
    except (requests.RequestException, json.JSONDecodeError) as e:
        print(f"Failed to retrieve reviews for page {page}. Error: {str(e)}")
        break

    if not data['reviews']:
        break

    for r in data['reviews']:
        review_text = r['content']
        author = r['author']['name']
        date = r['date']
        score = r['rating']
        reviews.append({"author": author, "date": date, "review_text": review_text, "score": score})

    page += 1

with open('reviews.json', 'w') as f:
    json.dump(reviews, f, indent=4)

你的评论URL都错了。
要获得评论，您需要以下部分，例如：

https://www.emag.ro/product-feedback/
covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM
/reviews/list
这给了你一个JSON，里面有你需要的一切。你自己看看：
https://www.emag.ro/product-feedback/covor-kring-meknes-1200-gsm-100-poliester-160x230-cm-maro-e2020-8b/pd/D605NYMBM/reviews/list
下面是一个完整的工作示例：

import json
import random
import re
import time

import requests

SAMPLE_URLS = [
    "https://www.emag.ro/covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM/?ref=profiled_categories_home_1_3&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50",
    "https://www.emag.ro/covor-kring-meknes-1200-gsm-100-poliester-160x230-cm-maro-e2020-8b/pd/D605NYMBM/?ref=profiled_categories_home_1_2&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50",
    "https://www.emag.ro/antiderapant-pentru-covor-tip-plasa-cali-poliester-80-x-180-cm-j119c9-99st4951/pd/DZDZXTMBM/?ref=profiled_categories_home_1_1&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50"
]

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42",
}

PRODUCT_ID = re.compile(r"pd/(.*?)/reviews")

def splitter(url: str) -> str:
    return url.rsplit('?')[0].split('ro/')[-1]

def save_json(data: dict, filename: str) -> None:
    with open(filename, 'w') as file:
        json.dump(data, file, indent=4)

review_urls = [
    f"https://www.emag.ro/product-feedback/{splitter(url)}reviews/list"
    for url in SAMPLE_URLS
]

with requests.Session() as s:
    s.headers.update(HEADERS)
    for url in review_urls:
        review_data = s.get(url).json()
        save_json(review_data, f"{re.search(PRODUCT_ID, url).group(1)}.json")
        print(url)
        for review in review_data["reviews"]["items"]:
            print(f"{review['rating']}: {review['title']}")
            print(f"{review['content']}")

这应该打印出来（为简洁起见缩短）：

https://www.emag.ro/product-feedback/covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM/reviews/list
2: Dezamagitor
A ajuns foarte repede, bine ambalat. Impaturit, f subtire, l-as numi paturica( voi anexa poze) se vede prin el asa este de subtire.
1: Nu recomand
Când am deschis easy boxul, pentru că acolo a sosit, am rămas surprins sa vad cat e se mic pachetul. Că o pătură. Nici pe departe grosimea de 4 cm. Numai aspect de covor nu are. Nu recomand.
1: Nemultumit
Cel mai prost produs comandat vreodată, nu se poate numi covor pt ca nu este, este ca o pătura! Oribil! O mare reclamație din partea mea merita acest produs! Foarte nemulțumita! Un mare retur! Timp pierdut degeaba!
1: Nu recomand
Nu este deloc nici pe departe cu ce este în poza... Asta nu e covor ci pătură de picnic
1: Nu recomand
The worst material ever
1: Nu recomand
Nu-i ce am văzut în poză îmi pare rău că l-am comandat
1: Nu recomand
Dezamăgitor, nu se poate numii covor ci mai degrabă o pătură, jalnic
1: Nu recomand
Nu se poate numi covor.
1: Nu recomand
Covorul este mai degrabă o paturica... Nu este deloc ca în poza și în descriere. Regret ca l-am comandat
1: Nu recomand
E efectiv o patura (pufoasa ce-i drept)

...

这也会在遍历URLs时将JSON响应转储到文件中。
示例JSON太大，无法显示所有内容，但运行以下代码，例如：

$ jq -r '.reviews.items[0].product.image' < D605NYMBM.json

应输出：

{
  "original": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg",
  "resized_images": [
    {
      "size": "80x80",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=80&height=80&hash=F675BCF7E208EEA9E59AAC0EFB10F96D"
    },
    {
      "size": "130x130",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=100&height=100&hash=2EED3E66E2550EE01608A7A622C72BBD"
    },
    {
      "size": "300x300",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=300&height=300&hash=AAD99FCED7B7A35CC54810994A4EA9B8"
    },
    {
      "size": "450x450",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=450&height=450&hash=21D2141F4007A320F28E893E5E299114"
    }
  ]
}

json 抓取评论的URL地址

1条答案

相关问题

热门标签

最新问答