Selenium Scrapy在无头模式下仍然打开Chrome浏览器

uelo1irk  于 2023-04-03  发布在  Go
关注(0)|答案(1)|浏览(208)

问题

1.我试图运行一个Selenium Scrapy Scrapy scrapy在无头模式(代码如下)
1.刮刀工作正常,在'头'模式,即与打开Chrome浏览器
1.当我从here添加指令并再次运行时,scraper就像什么都没改变一样运行。也就是说,它像以前一样运行,并打开Chrome
1.在Windows机器上工作。Chrome浏览器版本111
我应该改变什么才能让它在无头模式下运行?所有的建议都非常感谢。谢谢!!

代码

import scrapy
from scrapy_selenium import SeleniumRequest
import gspread
import scrapy 
from selenium import webdriver 
from selenium.webdriver.chrome.options import Options

ChromeOptions options = new ChromeOptions()
options.addArguments("--headless")

gc = gspread.service_account(filename = 'credentials2.json')
sh = gc.open_by_key('api_key').sheet1

class QuoteItem(scrapy.Item):
    # define the fields for your item here like:
    text = scrapy.Field()
    author = scrapy.Field()
    tags = scrapy.Field()

class QuotesSpider(scrapy.Spider):
    name = 'techleapsesc'

    def start_requests(self):
        url = 'https://finder.techleap.nl/investors.accelerators'
        yield SeleniumRequest(url=url, callback=self.parse, wait_time= 3)

    def parse(self, response):
        print("Line 24 - inside parse function")
        quote_item = QuoteItem()
        
        print("Line 27 - before for loop")
        print(response.css('div'))
        for quote in response.css('div.quote'):
            quote_item['text'] = quote.css('span.text::text').get()
            quote_item['author'] = quote.css('small.author::text').get()
            quote_item['tags'] = quote.css('div.tags a.tag::text').getall()
            self.sh.append_row(list(quote.values()))
            print(quote)
            yield quote_item

尝试了调用无头模式的不同方法

pieyvz9o

pieyvz9o1#

更改以下内容(Java代码):

ChromeOptions options = new ChromeOptions()
options.addArguments("--headless")

目标(Python代码):

options= webdriver.ChromeOptions()
options.add_argument("--headless")

相关问题