Scrapy按照所有的链接并获得状态

yyyllmsg 于 2022-12-29 发布在其他

关注(0)|答案(1)|浏览(146)

我想跟随网站的所有链接，并获得每个链接的状态像404，200。我尝试了这个：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

class someSpider(CrawlSpider):
  name = 'linkscrawl'
  item = []
  allowed_domains = ['mysite.com']
  start_urls = ['//mysite.com/']

  rules = (Rule (LinkExtractor(), callback="parse_obj", follow=True),
  )

  def parse_obj(self,response):
    item = response.url
    print(item)

我可以在控制台上看到没有状态代码的链接，如下所示：

mysite.com/navbar.html
mysite.com/home
mysite.com/aboutus.html
mysite.com/services1.html
mysite.com/services3.html
mysite.com/services5.html

但如何保存在文本文件中的所有链接的状态？

scrapy

来源：https://stackoverflow.com/questions/50200752/scrapy-follow-all-the-links-and-get-status

1条答案

按热度按时间

35g0bw711#

我解决了这个如下。希望这将有助于任何人谁需要。

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

class LinkscrawlItem(scrapy.Item):
    # define the fields for your item here like:
    link = scrapy.Field()
    attr = scrapy.Field()

class someSpider(CrawlSpider):
  name = 'linkscrawl'
  item = []

  allowed_domains = ['mysite.com']
  start_urls = ['//www.mysite.com/']

  rules = (Rule (LinkExtractor(), callback="parse_obj", follow=True),
  )

  def parse_obj(self,response):
    item = LinkscrawlItem()
    item["link"] = str(response.url)+":"+str(response.status)
    filename = 'links.txt'
    with open(filename, 'a') as f:
      f.write('\n'+str(response.url)+":"+str(response.status)+'\n')
    self.log('Saved file %s' % filename)

赞(0）回复(0）举报 2022-12-29

我来回答

Scrapy按照所有的链接并获得状态

1条答案

相关问题

热门标签

最新问答