scrapy 为什么我的scraper在运行时返回空结果集？

xghobddn  于 2023-03-18  发布在  其他
关注(0)|答案(2)|浏览(161)

当我运行此代码块时，它成功运行，但name字段返回空结果。请帮助我检查，我错过了什么？
这是我的密码

import scrapy

class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):

          containers = response.css('div.col-md-4') 

          for container in containers:
    
          yield
          {
             'name': container.css('a::text').get(),
          }

这是我得到的结果

2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Closing spider (finished)
2023-02-01 19:17:42 [scrapy.extensions.feedexport] INFO: Stored json feed (55 items) in: truck.json
2023-02-01 19:17:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 265,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 4912,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 1.916927,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2023, 2, 1, 18, 17, 42, 902380),
 'httpcompression/response_bytes': 22809,
 'httpcompression/response_count': 1,
 'item_scraped_count': 55,
 'log_count/DEBUG': 57,
 'log_count/INFO': 11,
 'log_count/WARNING': 1,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2023, 2, 1, 18, 17, 40, 985453)}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Spider closed (finished)
scrapy

2条答案

按热度按时间
elcex8rz

elcex8rz1#

您得到的结果集为空，因为CSS选择器div.col-md-4的方式不正确，而是使用[class="col-md-4 column"]并获取所需的输出。

import scrapy

class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):
        containers = response.css('[class="col-md-4 column"]')
        for container in containers:
            yield {
                'name': container.css('a::text').get()}

输出：

{'name': 'Alabama Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Alaska Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Arizona Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Arkansas Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'California Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Colorado Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Connecticut Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Delaware Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Florida Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Georgia Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Hawaii Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Idaho Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Illinois Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Indiana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Iowa Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Kansas Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Kentucky Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Louisiana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Maine Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Massachusetts Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Maryland Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Michigan Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Minnesota Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Mississippi Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Missouri Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Montana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Nevada Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Nebraska Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>

运行下面的代码而不使用scrapy项目

from scrapy.crawler import CrawlerProcess

import scrapy

class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):
        containers = response.css('[class="col-md-4 column"]')
        for container in containers:
            yield {
                'name': container.css('a::text').get()}

if __name__ == "__main__":  
    process = CrawlerProcess()
    process.crawl(TruckspiderSpider)
    process.start()
赞(0）分享  回复(0）举报  2023-03-18
bjg7j2ky

bjg7j2ky2#

@Md. Fazlul Hoque感谢您的时间和回复。
我终于找到了解决办法。我让我的本地计算机访问这个文件：2023-02-02 10:54:17 [filelock] DEBUG: Attempting to acquire lock 2733195998640 on C:\Users\ChiNedu\AppData\Local\Programs\Python\Python310\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock在我的AppData文件夹里，它像魔法一样工作。太高兴了。

赞(0）分享  回复(0）举报  2023-03-18
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com