我目前已经写了一个代码,它有多个线程(例如我使用了50个线程),对于每个线程,只允许一个代理在其中一个线程中(意味着一个代理不能在两个线程中)。
import contextlib
import random
import threading
import time
import requests
my_proxies = [
'http://140.99.107.100:2100',
'http://140.99.107.101:2100',
'http://140.99.107.102:2100',
'http://140.99.107.103:2100',
'http://140.99.107.104:2100',
'http://140.99.107.105:2100',
'http://140.99.107.106:2100',
'http://140.99.107.107:2100',
'http://140.99.107.108:2100',
'http://140.99.107.109:2100',
'http://140.99.107.110:2100',
'http://140.99.107.111:2100',
'http://140.99.107.112:2100',
'http://140.99.107.113:2100',
'http://140.99.107.114:2100',
'http://140.99.107.115:2100',
'http://140.99.107.116:2100',
'http://140.99.107.117:2100',
'http://140.99.107.118:2100',
'http://140.99.107.119:2100',
'http://140.99.107.120:2100',
'http://140.99.107.121:2100',
'http://140.99.107.122:2100',
]
# --------------------------------------------------------------------------- #
class AvailableProxiesManager:
_proxy_lock: threading.Lock = threading.Lock()
def __init__(self):
self._proxy_dict = dict.fromkeys(my_proxies, True)
@property
@contextlib.contextmanager
def proxies(self):
"""
Context manager that yields a random proxy from the list of available proxies.
:return: dict[str, str] - A random proxy.
"""
proxy = None
with self._proxy_lock:
while not proxy:
if available := [att for att, value in self._proxy_dict.items() if value]:
proxy = random.choice(available)
self._proxy_dict[proxy] = False
else:
print('Waiting ... no proxies available')
time.sleep(.2)
yield proxy
self._proxy_dict[proxy] = True # Return the proxy to the list of available proxies
# --------------------------------------------------------------------------- #
available_proxies = AvailableProxiesManager()
def main():
while True:
with available_proxies.proxies as proxy:
response = requests.get('https://httpbin.org/ip', proxies={'https': proxy})
if response.status_code == 403:
print('Lets put proxy on cooldown for 10 minutes and try with new one!')
time.sleep(120)
if __name__ == '__main__':
threads = []
for i in range(50):
t = threading.Thread(target=main)
threads.append(t)
t.start()
time.sleep(1)
然而,我的问题是,目前对于正在进行的每一个while True,它都使用一个新的随机代理,而我试图实现的是,我希望在同一个线程中使用同一个代理,直到响应状态为403。这意味着,在开始时,如果线程1获得代理:http://140.99.107.100:2100
,则它应该在线程1中使用,直到它得到403。
我的问题是,我如何才能使同一个代理被使用,直到它命中响应403?
预期:
- 403之前的代理相同 *
实际: - 为每个GET请求新建代理 *
1条答案
按热度按时间oyt4ldly1#
如果您停止使用上下文管理器,(remove @contextlib.contextmanager)并执行以下操作,会怎样:
希望能有所帮助,祝你好运!