python-3.x 如何使用相同的代理直到特定的响应状态?

sirbozc5  于 2022-12-05  发布在  Python
关注(0)|答案(1)|浏览(118)

我目前已经写了一个代码,它有多个线程(例如我使用了50个线程),对于每个线程,只允许一个代理在其中一个线程中(意味着一个代理不能在两个线程中)。

import contextlib
import random
import threading
import time

import requests

my_proxies = [
    'http://140.99.107.100:2100',
    'http://140.99.107.101:2100',
    'http://140.99.107.102:2100',
    'http://140.99.107.103:2100',
    'http://140.99.107.104:2100',
    'http://140.99.107.105:2100',
    'http://140.99.107.106:2100',
    'http://140.99.107.107:2100',
    'http://140.99.107.108:2100',
    'http://140.99.107.109:2100',
    'http://140.99.107.110:2100',
    'http://140.99.107.111:2100',
    'http://140.99.107.112:2100',
    'http://140.99.107.113:2100',
    'http://140.99.107.114:2100',
    'http://140.99.107.115:2100',
    'http://140.99.107.116:2100',
    'http://140.99.107.117:2100',
    'http://140.99.107.118:2100',
    'http://140.99.107.119:2100',
    'http://140.99.107.120:2100',
    'http://140.99.107.121:2100',
    'http://140.99.107.122:2100',
]

# --------------------------------------------------------------------------- #

class AvailableProxiesManager:
    _proxy_lock: threading.Lock = threading.Lock()

    def __init__(self):
        self._proxy_dict = dict.fromkeys(my_proxies, True)

    @property
    @contextlib.contextmanager
    def proxies(self):
        """
        Context manager that yields a random proxy from the list of available proxies.

        :return: dict[str, str] - A random proxy.
        """
        proxy = None

        with self._proxy_lock:
            while not proxy:
                if available := [att for att, value in self._proxy_dict.items() if value]:
                    proxy = random.choice(available)
                    self._proxy_dict[proxy] = False
                else:
                    print('Waiting ... no proxies available')
                    time.sleep(.2)

        yield proxy
        self._proxy_dict[proxy] = True  # Return the proxy to the list of available proxies

# --------------------------------------------------------------------------- #

available_proxies = AvailableProxiesManager()

def main():
    while True:
        with available_proxies.proxies as proxy:
            response = requests.get('https://httpbin.org/ip', proxies={'https': proxy})

            if response.status_code == 403:
                print('Lets put proxy on cooldown for 10 minutes and try with new one!')

        time.sleep(120)

if __name__ == '__main__':
    threads = []
    for i in range(50):
        t = threading.Thread(target=main)
        threads.append(t)
        t.start()
        time.sleep(1)

然而,我的问题是,目前对于正在进行的每一个while True,它都使用一个新的随机代理,而我试图实现的是,我希望在同一个线程中使用同一个代理,直到响应状态为403。这意味着,在开始时,如果线程1获得代理:http://140.99.107.100:2100,则它应该在线程1中使用,直到它得到403。
我的问题是,我如何才能使同一个代理被使用,直到它命中响应403?

预期

  • 403之前的代理相同 *
    实际
  • 为每个GET请求新建代理 *
oyt4ldly

oyt4ldly1#

如果您停止使用上下文管理器,(remove @contextlib.contextmanager)并执行以下操作,会怎样:

def main():
    proxy = next(available_proxies.proxies)
    while True:
        response = requests.get('https://httpbin.org/ip', proxies={'https': proxy})
        if response.status_code == 403:
            proxy = next(available_proxies.proxies)

        time.sleep(120)

希望能有所帮助,祝你好运!

相关问题