问题
1.我试图运行一个Selenium Scrapy Scrapy scrapy在无头模式(代码如下)
1.刮刀工作正常,在'头'模式,即与打开Chrome浏览器
1.当我从here添加指令并再次运行时,scraper就像什么都没改变一样运行。也就是说,它像以前一样运行,并打开Chrome
1.在Windows机器上工作。Chrome浏览器版本111
我应该改变什么才能让它在无头模式下运行?所有的建议都非常感谢。谢谢!!
代码
import scrapy
from scrapy_selenium import SeleniumRequest
import gspread
import scrapy
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
ChromeOptions options = new ChromeOptions()
options.addArguments("--headless")
gc = gspread.service_account(filename = 'credentials2.json')
sh = gc.open_by_key('api_key').sheet1
class QuoteItem(scrapy.Item):
# define the fields for your item here like:
text = scrapy.Field()
author = scrapy.Field()
tags = scrapy.Field()
class QuotesSpider(scrapy.Spider):
name = 'techleapsesc'
def start_requests(self):
url = 'https://finder.techleap.nl/investors.accelerators'
yield SeleniumRequest(url=url, callback=self.parse, wait_time= 3)
def parse(self, response):
print("Line 24 - inside parse function")
quote_item = QuoteItem()
print("Line 27 - before for loop")
print(response.css('div'))
for quote in response.css('div.quote'):
quote_item['text'] = quote.css('span.text::text').get()
quote_item['author'] = quote.css('small.author::text').get()
quote_item['tags'] = quote.css('div.tags a.tag::text').getall()
self.sh.append_row(list(quote.values()))
print(quote)
yield quote_item
尝试了调用无头模式的不同方法
1条答案
按热度按时间pieyvz9o1#
更改以下内容(Java代码):
目标(Python代码):