我试图刮一个网站，其中部分呈现内容使用JS。
我找到了这个项目：https://github.com/scrapinghub/sample-projects/tree/master/splash_smart_proxy_manager_example，它非常清楚地解释了如何设置。下面是我现在拥有的内容：
坞缀：

version: '3.8'

services:
    scraping:
        build:
            context: .
            dockerfile: Dockerfile
        volumes:
            - "./scraping:/scraping"
        environment:
            - PYTHONUNBUFFERED=1
        depends_on:
            - splash
        links:
            - splash
    splash:
        image: scrapinghub/splash
        restart: always
        expose:
            - 5023
            - 8050
            - 8051
        ports:
            - "5023:5023"
            - "8050:8050"
            - "8051:8051"

蜘蛛：

class HappySider(scrapy.Spider):
    ...
    custom_settings = {
        'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
        'SPIDER_MIDDLEWARES': {
            'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
        },
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy_splash.SplashCookiesMiddleware': 723,
            'scrapy_splash.SplashMiddleware': 725,
            'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
        },
        'ITEM_PIPELINES': {
            'scraping.pipelines.HappySpiderPipeline': 300,
        },
        'RETRY_HTTP_CODES': [500, 502, 503, 504, 522, 524, 408, 429, 403],
        'RETRY_TIMES': 20,
        'DOWNLOAD_DELAY': 5,
        'DOWNLOAD_TIMEOUT': 30,
        'CONCURRENT_REQUESTS': 1,
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'COOKIES_ENABLED': False,
        'ROBOTSTXT_OBEY': True,
        # enable Zyte Proxy
        'ZYTE_SMARTPROXY_ENABLED': True,
        # the APIkey you get with your subscription
        'ZYTE_SMARTPROXY_APIKEY': '<my key>',
        'SPLASH_URL': 'http://splash:8050/',
    }

    def __init__(self, testing=False, name=None,**kwargs):
        self.LUA_SOURCE = get_data(
            'scraping', 'scripts/smart_proxy_manager.lua'
        ).decode('utf-8')
        super().__init__(name,**kwargs)

    def start_requests(self):

        yield SplashRequest(
            url='https://www.someawesomesi.te',
            endpoint='execute',
            args={
                'lua_source': self.LUA_SOURCE,
                'crawlera_user': self.settings['ZYTE_SMARTPROXY_APIKEY'],
                'timeout': 90,
            },
            # tell Splash to cache the lua script, to avoid sending it for every request
            cache_args=['lua_source'],
            meta={
                'max_retry_times': 10,
            },
            callback=self.my_callback
        )

我得到的输出是：

2022-08-10 13:09:32 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.someawesomesi.te via http://splash:8050/execute> (failed 1 times): 504 Gateway Time-out

我不知道如何继续。我确实查了一下为什么它会给我504，splash docks确实建议了一些处理它的方法...但是我没有太多的并发URL，脚本在第一个失败了。另外，我正在刮的网站非常快，如果我只是使用Zyte而不使用splash，那么它刮得非常快。
因此，如果有人能建议这里有什么问题，以及如何解决它-我将非常感谢。