我正在创建一个脚本,列出所有业务从一个网站,它需要的刮(姓名,地址,网站,电子邮件,电话号码)。我得到的一部分,我有点可以刮电子邮件,但我有一个小问题,我不能只是告诉我的脚本采取所有他们,他们是specyifc和需要包含[Biuro或秘书处或名称部分的网站www.(namePart).com]和我有点不知道如何做到这一点。这里是我的代码:
# -*- coding: utf-8 -*-
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import Request, Spider
class RynekMainSpider(scrapy.Spider):
name = "RynekMain"
start_urls = [
'https://rynekpierwotny.pl/deweloperzy/?page=1']
def parse(self, response):
websites = response.css('div#root')[0]
PAGETEST = response.xpath('//a[contains(@class,"rp-173nt6g")]/../following-sibling::li').css('a::attr(href)').get()
for website in websites.css('li.rp-np9kb1'):
page = website.css('a::attr(href)').get()
address = website.css('address.rp-o9b83y::text').get()
name = website.css('h2.rp-69f2r4::text').get()
params = {
'address' : address,
'name' : name,
'href' : page,
}
url = response.urljoin(page)
yield Request(url=url, cb_kwargs={'params': params}, callback=self.parseMain)
yield Request(url=response.urljoin(PAGETEST), callback=self.parse)
def parseMain(self, response, params=None):
# print(response.url)
website = response.css('div.rp-l0pkv6 a::attr(href)').get()
params['website'] = website
urlem = response.urljoin(website)
yield Request(url=urlem, cb_kwargs={'params': params}, callback=self.parseEmail)
def parseEmail(self,response, params=None):
email = response.xpath('//a[contains(@href, "@")]/@href').get()
params['email'] = email
yield params
if __name__ == "__main__":
process =CrawlerProcess()
process.crawl(RynekMainSpider)
process.start()
提前感谢您的帮助!
1条答案
按热度按时间huus2vyu1#
在
parseEmail
方法中,提取电子邮件地址后,只需像检查任何字符串一样检查提取的字符串。例如