python 用BeautifulSoup刷网页

0md85ypi 于 2022-12-17 发布在 Python

关注(0)|答案(2)|浏览(112)

我试图刮这个网站：https://www.senate.gov/general/contact_information/senators_cfm.cfm
我的代码：

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

print(soup)

问题是它实际上并没有进入网站，我在soup var中得到的HTML与正确网页中的HTML完全不同。

python

来源：https://stackoverflow.com/questions/62087763/scraping-webpage-using-beautifulsoup

2条答案

按热度按时间

ldfqzlk81#

这对我很有效

headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    }
r = requests.get(URL,headers=headers)

在此处找到信息-https://towardsdatascience.com/5-strategies-to-write-unblock-able-web-scrapers-in-python-5e40c147bdaf

赞(0）回复(0）举报 2022-12-17

i2loujxw2#

复制HTTP 503 Error while using python requests module
试试看：

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'

page = requests.post(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

赞(0）回复(0）举报 2022-12-17

我来回答

python 用BeautifulSoup刷网页

2条答案

相关问题

热门标签

最新问答