如何设置一个授权的Scrapy代理?

s1ag04yj  于 2022-11-09  发布在  其他
关注(0)|答案(2)|浏览(114)

我的中间件设置:

from w3lib.http import basic_auth_header 
class CustomProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "111.11.11.111:1111"
        request.headers['Proxy - Authorization'] = basic_auth_header('login', 'password')

我的设置:

DOWNLOADER_MIDDLEWARES = {
    'my_project.middlewares.CustomProxyMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

启动后,我收到一个错误:

scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 217.29.53.106:51725 [{'status': 407, 'reason': b'Proxy Authentication Required'}]

原因是什么,如何修复它?(我使用有效的https代理)

kupeojn6

kupeojn61#

尝试将标头名称更改为Proxy-Authorization

request.headers['Proxy-Authorization'] = basic_auth_header('login', 'password')
ht4b089n

ht4b089n2#

proxy = [
  'http': 'http://{user}:{password}@{host}:{port}',
  'https': 'https://{user}:{password}@{host}:{port}',
]

yield scrapy.request(url=url, proxy=proxy}

这个不管用吗?

相关问题