我目前正在使用Python中的selenium进行网络抓取在线数据库。数据库的格式需要在页面之间导航，以便抓取我感兴趣的数据，每次我运行代码时，我总是会遇到502 Bad Gateway Error（下图）。

这个错误消息似乎会消失 * 有时 *，但它似乎取决于这个502在循环中弹出的位置。任何关于如何避免这种情况的建议将不胜感激。我还附上了下面与Chrome交互的部分代码以供参考：

# ! Final !
#### Define Driver & Starting URL ####
# Location of chromedriver
driver_path = "/Users/shrey/Desktop/Python Projects/Selenium/chromedriver"

# Beginning url & initialize driver
url = "https://tamu.libguides.com/az.php"
driver = webdriver.Chrome()

# Make driver wait for elements to load when find_element() is run for the rest of our code
driver.implicitly_wait(10)

# Launch driver
driver.get(url)

# Press "Ancestry Database" link
driver.find_element(By.LINK_TEXT,
                    "Ancestry Library").click()

# Give time for user to login to database
time.sleep(30)

# Go to link where we can search from
home = "https://www.ancestrylibrary.com/search/collections/1742/"
driver.get(home)

# Switch to first tab (Search tab we just opened)
driver.switch_to.window(driver.window_handles[0])

#### Loop through each year present in the data ####
for yr in range(1886, 1952):
    # Go to search home
    driver.get(home)
    
    # Find textbox & Input Year --------
    year_input = driver.find_element(By.CSS_SELECTOR, "#sfs_SelfCivilYear")
    year_input.send_keys(str(yr))

    # Press "search" button
    driver.find_element(By.CSS_SELECTOR, "#searchButton").click()

    # Determine number of times we need to loop --------
    # Find text which includes total number of results (formatted as "Results 1–20 of 1,351")
    n_raw = driver.find_element(By.XPATH,
                                '//*[@id="results-header"]/h3').text

    # Isolate the important number (1,351)
    n_num = (tot_results.split()[-1]) # pulls the last word from the string - our desired number

    # Remove comma and convert to number ("1,351" >>> 1351)
    n_total = int(re.sub(",", "", n_num))

    # Determine number of loops we need to do to scrape all the data
    loop_count = math.floor(n_total/20) + 1

    # Loop thru pages and collect links --------
    # Init empty list
    links = []
    
    # Loop n times (calc'd earlier)
    for i in range(loop_count):
        
        # If we are on our last iter, do the same but do not click "next page" button
        if i == range(loop_count)[-1]: 
            # Find & Store all "View Result" links
            current_pg_links = driver.find_elements(By.CSS_SELECTOR, 
                                                    ".srchFoundDB a")

            # Loop through all links pulled & append
            for link in current_pg_links:
                # Get actual url from 'href' attribute
                url = link.get_attribute('href')

                # Append URL to final list
                links.append(url)

        else:
            # Find & Store all "View Result" links
            current_pg_links = driver.find_elements(By.CSS_SELECTOR, 
                                                    ".srchFoundDB a")

            for link in current_pg_links:
                # Get actual url from 'href' attribute
                url = link.get_attribute('href')

                # Append URL to final list
                links.append(url)

            # Press "next page" button
            driver.find_element(By.CSS_SELECTOR,
                                "a.ancBtn.sml.green.icon.iconArrowRight").click()