scrapy 收到带有特定文本的电子邮件很不方便

wfypjpf4  于 2022-11-09  发布在  其他
关注(0)|答案(1)|浏览(116)

我正在创建一个脚本,列出所有业务从一个网站,它需要的刮(姓名,地址,网站,电子邮件,电话号码)。我得到的一部分,我有点可以刮电子邮件,但我有一个小问题,我不能只是告诉我的脚本采取所有他们,他们是specyifc和需要包含[Biuro或秘书处或名称部分的网站www.(namePart).com]和我有点不知道如何做到这一点。这里是我的代码:


# -*- coding: utf-8 -*-

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import Request, Spider

class RynekMainSpider(scrapy.Spider):
    name = "RynekMain"
    start_urls = [
        'https://rynekpierwotny.pl/deweloperzy/?page=1']
    def parse(self, response):
        websites = response.css('div#root')[0]
        PAGETEST = response.xpath('//a[contains(@class,"rp-173nt6g")]/../following-sibling::li').css('a::attr(href)').get()
        for website in websites.css('li.rp-np9kb1'):
            page = website.css('a::attr(href)').get()
            address = website.css('address.rp-o9b83y::text').get()
            name = website.css('h2.rp-69f2r4::text').get()
            params = {
            'address' : address,
            'name' : name,
            'href' : page,
            }
            url  = response.urljoin(page)

            yield Request(url=url, cb_kwargs={'params': params}, callback=self.parseMain)

        yield Request(url=response.urljoin(PAGETEST), callback=self.parse)

    def parseMain(self, response, params=None):
        # print(response.url)
        website = response.css('div.rp-l0pkv6 a::attr(href)').get()
        params['website'] = website
        urlem = response.urljoin(website)
        yield Request(url=urlem, cb_kwargs={'params': params}, callback=self.parseEmail)

    def parseEmail(self,response, params=None):
        email = response.xpath('//a[contains(@href, "@")]/@href').get()
        params['email'] = email        
        yield params
if __name__ == "__main__":
    process =CrawlerProcess()
    process.crawl(RynekMainSpider)
    process.start()

提前感谢您的帮助!

huus2vyu

huus2vyu1#

parseEmail方法中,提取电子邮件地址后,只需像检查任何字符串一样检查提取的字符串。
例如

from urllib.parse import urlsplit

def parseEmail(self,response, params=None):
    email = response.xpath('//a[contains(@href, "@")]/@href').get()
    netloc = urlsplit(response.url).netloc
    if 'Biuro' in email or 'Sekretariat' in email:
        params['email'] = email
    elif any([(i in email) for i in netloc.split('.')[:-1] if i != 'www']):
        params['email'] = email
    yield params

相关问题