R语言 Web抓取在传递不同的标头后返回403错误

wgeznvg7 于 2023-01-03 发布在其他

关注(0)|答案(1)|浏览(328)

我试图刮一个网站使用的一个软件包在R。
当我运行以下命令时：

library(idealisto) #https://github.com/hmeleiro/idealisto
get_city("https://www.idealista.com/alquiler-viviendas/madrid-madrid/", "sale")

我得到：
read_html. response（.）中的错误：禁止访问（HTTP 403）。
查看函数get_city()的更多细节，我发现问题出在以下代码部分：

desktop_agents <- c("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0")

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"
x <- GET(url, add_headers(`user-agent` = desktop_agents[sample(1:10, 1)]))

它返回以下输出：
回复[https：//www.idealista.com/en/venta-viviendas/madrid-provincia/]
日期：2022年4月4日18：52状态：403内容类型：应用程序/json;字符集= utf-8大小：360乙
然而，我应该得到一个Status: 200。我试图手动传递一些headers，但我仍然得到相同的Status错误：

headers = c(
  'accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'accept-encoding' = 'gzip, deflate, br',
  'accept-language' = 'es-ES,es;q=0.9,en;q=0.8',
  'cache-control' = 'max-age=0',
  'referer' = 'https://www.idealista.com/en/',
  'sec-fetch-mode' = 'navigate',
  'sec-fetch-site' = 'same-origin',
  'sec-fetch-user' = '?1',
  'upgrade-insecure-requests' =  '1',
  'user-agent' = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'
)

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"
x <- GET(url, add_headers(headers))

知道如何解决这个Status错误吗？

r

来源：https://stackoverflow.com/questions/71742402/web-scraping-returning-a-403-error-after-passing-different-headers

1条答案

按热度按时间

nbnkbykc1#

你的add_headers语法错误，你不能传递一个命名向量，你必须直接传递参数给函数：

library(httr)

headers <- add_headers(
  'accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'accept-encoding' = 'gzip, deflate, br',
  'accept-language' = 'es-ES,es;q=0.9,en;q=0.8',
  'cache-control' = 'max-age=0',
  'referer' = 'https://www.idealista.com/en/',
  'sec-fetch-mode' = 'navigate',
  'sec-fetch-site' = 'same-origin',
  'sec-fetch-user' = '?1',
  'upgrade-insecure-requests' =  '1',
  'user-agent' = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'
)

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"

GET(url, headers)
#> Response [https://www.idealista.com/en/venta-viviendas/madrid-provincia/]
#>   Date: 2022-04-04 19:10
#>   Status: 200
#>   Content-Type: text/html; charset=UTF-8
#>   Size: 263 kB
#> <!DOCTYPE html>
#> <html lang="en" env="es" username="" data-userauth="false" class="">
#> <head>
#> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
#> <title>Property for sale in Madrid province, Spain: houses and flats &#8212; ...
#> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
#> <meta name="description" content="37,980 houses and flats for sale in Madrid,...
#> <meta name="author" content="idealista.com">
#> <meta http-equiv="cleartype" content="on">
#> <meta name="pragma" content="no-cache"/>
#> ...

由reprex package（v2.0.1）于2022年4月4日创建

赞(0）回复(0）举报 2023-01-03

我来回答

R语言 Web抓取在传递不同的标头后返回403错误

1条答案

相关问题

热门标签

最新问答