我尝试从网站https://www.etf.com/channels抓取ETF。然而，无论我尝试什么，当我尝试访问它时，它返回503错误。我尝试使用不同的用户代理和头文件，但它仍然不让我访问它。有时当我尝试通过浏览器访问网站时，弹出一个页面，“检查连接是否安全”所以我认为他们已经有了一些东西来停止刮擦。我看到其他人问同样的问题，答案总是说添加一个用户代理，但这对这个网站不起作用。
斯克拉皮

class BrandETFs(scrapy.Spider):
    name = "etfs"
    start_urls = ['https://www.etf.com/channels']

    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "en-US,en;q=0.5",
        "Connection": "keep-alive",
        "Host": "www.etf.com",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "cross-site",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0"
    }

    custom_settings = {'DOWNLOAD_DELAY': 0.3, "CONCURRENT_REQUESTS": 4}

    def start_requests(self):
        url = self.start_urls[0]
        yield scrapy.Request(url=url)

    def parse(self, response):
        test = response.css('div.discovery-slat')
        yield {
            "test": test
        }

请求数

import requests

url = 'https://www.etf.com/channels'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36',
    'Referer': 'https://google.com',
    'Origin': 'https://www.etf.com'
}
r = requests.post(url, headers=headers)
r.raise_for_status()

有办法绕过这些街区进入网站吗？