selenium 在Python中进行废弃时未加载网页

zengzsys  于 2022-11-24  发布在  Python
关注(0)|答案(1)|浏览(105)

我有一个数据集,其中包含的网址只是拨号网站,我试图提取一些信息,如卖方名称。下面我附上了一个样本数据

dict_test  = {"Id" : [1000, 1001, 1002],
             "Online_url" : ['https://www.justdial.com/Mumbai/Sunrise-Info-Solutions-Pvt-Ltd-Near-Airtel-Gallery/022PXX22-XX22-220719102528-J5Q2_BZDET?xid=TXVtYmFpIE1vYmlsZSBEZWFsZXJz',
                            'https://www.justdial.com/Mumbai/Riddhi-Siddhi-Mobile-Gallery-Electronic-Opposite-Jain-Plaza-Ambernath/022PXX22-XX22-210519191020-K2U6_BZDET?xid=TXVtYmFpIE1vYmlsZSBEZWFsZXJz',
                            'https://www.justdial.com/Mumbai/Bharat-Communication-Opposite-Vibgyor-School-Goregaon-West/022PXX22-XX22-130103150323-S4V9_BZDET?xid=TXVtYmFpIE1vYmlsZSBEZWFsZXJz']}
df_test = pd.DataFrame(dict_test)

下面的脚本是我用来提取卖方信息

options = webdriver.ChromeOptions() 
options.add_experimental_option("excludeSwitches", ['enable-automation'])
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--disable-notifications")
options.add_argument( "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36")
options.add_argument("--remote-debugging-port=9222")
driver = webdriver.Chrome(executable_path=r'C:\Users\admin\Downloads\chromedriver_lates\chromedriver.exe', options = options)
driver.maximize_window()
driver.implicitly_wait(10)
driver.get('https://www.justdial.com/')
time.sleep(2)

def webpage_extract(min_count, max_count, df_test, folder, file_name):
    for i in range(min_count,max_count):
        try:
            driver.set_page_load_timeout(5)
            driver.switch_to.window(driver.window_handles[0])
            driver.execute_script("window.open('');")
            # Switch to the new window and open new URL
            driver.switch_to.window(driver.window_handles[1])
            driver.get(df_test['Online_url'].iloc[i])
            time.sleep(5)
        except TimeoutException as ex:
            isrunning = 0
            print("Exception has been thrown.")

        try:
            myElem = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.CLASS_NAME, 'fn')))
        except TimeoutException:
            print("Loading took too much time!")
        
### Below command is used to close the Pop-up
        try:
            driver.find_element("xpath", '//*[@id="best_deal_detail_div"]/section/span').click()
            driver.find_element("xpath", '//*[@id="best_deal_detail_div"]/section/span').click()
        except:
            "Pop_Up"
            
        try:
            seller_info=driver.find_element("xpath", "/html/body/div[2]/div[1]/div/div[1]/div[2]/div/div/h1/span/span").text
            print("Seller_Name: ", seller_info)
        except:
            seller_info="Extraction_Error"
            
        print("Iteration {} : Information Extracted for Seller {}".format(i, seller_info))
        driver.delete_all_cookies()
        # Closing the tab
        driver.close()
        time.sleep(2)
    return None

%time webpage_extract(0, len(df_test), df_test, folder, file_name)

上述代码的问题是,第一个URL数据提取正确的信息,但第二个URL的网页没有加载,并显示有关:空白URL。有什么办法来解决这个问题或跳过这个URL,并移动到下一个URL?

ruarlubt

ruarlubt1#

不应在webpage_extract结束时关闭驱动程序。

相关问题