如何获得一个json响应与scrapy？

5n0oy7gb 于 2022-11-09 发布在其他

关注(0)|答案(1)|浏览(142)

我尝试用scrapy的默认请求库复制相同的代码块，因为它更快，更高效。我尝试从steamstore api中检索数据：

import requests
import json

url = "http://store.steampowered.com/api/appdetails/"
appid = 10
name = "Counter-Strike"
parameters = {"appids": appid}
response = requests.get(url=url, params=parameters)
json_data = response.json()
json_app_data = json_data[str(appid)]

if json_app_data['success']:
    data = json_app_data['data']
else:
    data = {'name': name, 'steam_appid': appid}

print(data)

scrapy

来源：https://stackoverflow.com/questions/73122752/how-to-get-a-json-response-with-scrapy

1条答案

按热度按时间

ifmq2ha21#

实际上，scrapy不知道params是什么？这意味着它不能将params作为参数。因此，另一种方法是，您可以将整个api url注入scrapy Request方法中。

范例：

import scrapy
import json
from scrapy.crawler import CrawlerProcess
class TestSpider(scrapy.Spider):
    name = 'test'
    def start_requests(self):  
        headers= {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'}
        api_url='https://store.steampowered.com/api/appdetails/?appids=10'
        yield scrapy.Request(
            url= api_url,
            method='GET',
            headers=headers,
            callback=self.parse
            )

    def parse(self, response):
        json_app_data = json.loads(response.body)

        print(json_app_data['10']['data'])

if __name__ == "__main__":
    process = CrawlerProcess(TestSpider)
    process.crawl()
    process.start()

输出：

{'type': 'game', 'name': 'Counter-Strike', 'steam_appid': 10, 'required_age': 0, 'is_free': False, 'detailed_description': "Play the world's number 
1 online action game. Engage in an incredibly realistic brand of terrorist warfare in this wildly popular team-based game. Ally with teammates to complete strategic missions. Take out enemy sites. Rescue hostages. Your role affects your team's success. Your team's success affects your role.", 'about_the_game': "Play the world's number 1 online action game. Engage in an incredibly realistic brand of terrorist warfare in this wildly popular team-based game. Ally with teammates to complete strategic missions. Take out enemy sites. Rescue hostages. Your role affects your team's success. Your team's success affects your role.", 'short_description': "Play the world's number 1 online action game. Engage in an incredibly realistic brand of terrorist warfare in this wildly popular team-based game. Ally with teammates to complete strategic missions. Take out enemy sites. Rescue hostages. Your role affects your team's success. Your team's success affects your role.", 'supported_languages': 'English<strong>*</strong>, French<strong>*</strong>, German<strong>*</strong>, Italian<strong>*</strong>, Spanish - Spain<strong>*</strong>, Simplified Chinese<strong>*</strong>, Traditional Chinese<strong>*</strong>, Korean<strong>*</strong><br><strong>*</strong>languages with full audio support', 'header_image': 'https://cdn.akamai.steamstatic.com/steam/apps/10/header.jpg?t=1602535893', 'website': None, 'pc_requirements': {'minimum': '\r\n\t\t\t<p><strong>Minimum:</strong> 500 mhz processor, 96mb ram, 16mb video card, Windows XP, Mouse, Keyboard, Internet Connection<br /></p>\r\n\t\t\t<p><strong>Recommended:</strong> 800 mhz processor, 128mb ram, 32mb+ video card, Windows XP, Mouse, Keyboard, Internet Connection<br /></p>\r\n\t\t\t'}, 'mac_requirements': {'minimum': 'Minimum: OS X  Snow Leopard 10.6.3, 1GB RAM, 4GB Hard Drive Space,NVIDIA GeForce 8 or higher, ATI X1600 or higher, or Intel HD 3000 or higher Mouse, 
Keyboard, Internet Connection'}, 'linux_requirements': {'minimum': 'Minimum: Linux Ubuntu 12.04, Dual-core from Intel or AMD at 2.8 GHz, 1GB Memory, nVidia GeForce 8600/9600GT, ATI/AMD Radeaon HD2600/3600 (Graphic Drivers: nVidia 310, AMD 12.11), OpenGL 2.1, 4GB Hard Drive Space, OpenAL Compatible Sound Card'}, 'developers': ['Valve'], 'publishers': ['Valve'], 'price_overview': {'currency': 'USD', 'initial': 549, 'final': 549, 'discount_percent': 0, 'initial_formatted': '', 'final_formatted': '$5.49 USD'}, 'packages': [574941, 7], 'package_groups': [{'name': 'default', 'title': 'Buy Counter-Strike', 'description': '', 'selection_text': 'Select a purchase option', 'save_text': '', 'display_type': 0, 'is_recurring_subscription': 'false', 'subs': [{'packageid': 7, 'percent_savings_text': ' ', 'percent_savings': 0, 'option_text': 'Counter-Strike: Condition Zero - $5.49 USD', 'option_description': '', 'can_get_free_license': '0', 'is_free_license': False, 'price_in_cents_with_discount': 549}, {'packageid': 574941, 'percent_savings_text': ' ', 'percent_savings': 0, 'option_text': 'Counter-Strike - Commercial License - $5.49 USD', 'option_description': '', 'can_get_free_license': '0', 'is_free_license': False, 'price_in_cents_with_discount': 549}]}], 'platforms': {'windows': True, 'mac': True, 'linux': True}, 'metacritic': {'score': 88, 'url': 'https://www.metacritic.com/game/pc/counter-strike?ftag=MCD-06-10aaa1f'}, 'categories': [{'id': 1, 'description': 'Multi-player'}, {'id': 49, 'description': 'PvP'}, {'id': 36, 'description': 'Online PvP'}, {'id': 37, 'description': 'Shared/Split Screen PvP'}, {'id': 8, 'description': 'Valve Anti-Cheat enabled'}], 'genres': [{'id': '1', 'description': 'Action'}], 'screenshots': [{'id': 0, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000132.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000132.1920x1080.jpg?t=1602535893'}, {'id': 1, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000133.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000133.1920x1080.jpg?t=1602535893'}, {'id': 2, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000134.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000134.1920x1080.jpg?t=1602535893'}, {'id': 3, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000135.600x338.jpg?t=1602535893', 
'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000135.1920x1080.jpg?t=1602535893'}, {'id': 4, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000136.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000000136.1920x1080.jpg?t=1602535893'}, {'id': 5, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002540.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002540.1920x1080.jpg?t=1602535893'}, {'id': 6, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002539.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002539.1920x1080.jpg?t=1602535893'}, {'id': 7, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002538.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002538.1920x1080.jpg?t=1602535893'}, {'id': 8, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002537.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002537.1920x1080.jpg?t=1602535893'}, {'id': 9, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002536.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002536.1920x1080.jpg?t=1602535893'}, {'id': 10, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002541.600x338.jpg?t=1602535893', 'path_full': 
'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002541.1920x1080.jpg?t=1602535893'}, {'id': 11, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002542.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002542.1920x1080.jpg?t=1602535893'}, {'id': 12, 'path_thumbnail': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002543.600x338.jpg?t=1602535893', 'path_full': 'https://cdn.akamai.steamstatic.com/steam/apps/10/0000002543.1920x1080.jpg?t=1602535893'}], 'recommendations': {'total': 122748}, 'release_date': {'coming_soon': False, 'date': '1 Nov, 2000'}, 'support_info': {'url': 'http://steamcommunity.com/app/10', 'email': ''}, 'background': 'https://cdn.akamai.steamstatic.com/steam/apps/10/page_bg_generated_v6b.jpg?t=1602535893', 
'background_raw': 'https://cdn.akamai.steamstatic.com/steam/apps/10/page_bg_generated.jpg?t=1602535893', 'content_descriptors': {'ids': [2, 5], 'notes': 'Includes intense violence and blood.'}}
2022-07-26 18:32:01 [scrapy.core.engine] INFO: Closing spider (finished)
2022-07-26 18:32:01 [scrapy.statscollectors] INFO: Dumping Scrapy stats:  
{'downloader/request_bytes': 324,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 2192,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1

赞(0）回复(0）举报 2022-11-09

我来回答

如何获得一个json响应与scrapy？

1条答案

相关问题

热门标签

最新问答