我遇到了同样的问题，并通过为sqlite3创建空模块来修复它，如以下答案所述：https://stackoverflow.com/a/44532317/5441099。显然，Scrapy导入了sqlite3，但并不一定要使用它。Python3希望主机上有sqlite3，但AWS Lambda机器上没有。日志中并不总是显示错误消息。
这意味着您可以通过切换到Python2来使其工作，或者像我一样为sqlite3创建空模块。
我运行爬虫的入口文件如下所示，它在Lambda上与Python3.6一起工作：

# run_crawler.py

# crawl() is invoked from the handler function in Lambda

import os
from my_scraper.spiders.my_spider import MySpider
from scrapy.crawler import CrawlerProcess

# Start sqlite3 fix

import imp
import sys
sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")

# End sqlite3 fix

def crawl():
    process = CrawlerProcess(dict(
        FEED_FORMAT='json',
        FEED_URI='s3://my-bucket/my_scraper_feed/' +
        '%(name)s-%(time)s.json',
        AWS_ACCESS_KEY_ID=os.getenv('AWS_ACCESS_KEY_ID'),
        AWS_SECRET_ACCESS_KEY=os.getenv('AWS_SECRET_ACCESS_KEY'),
    ))
    process.crawl(MySpider)
    process.start()  # the script will block here until all crawling jobs are finished

if __name__ == '__main__':
    crawl()

正如@viktorAndersen的回答解决了AWS Lambda中的崩溃/工作不符合预期的问题。
我有一个沉重的蜘蛛爬行2000网址，我面临2个问题
1.当我运行scrapy函数超过1次时，出现了ReactorNotRestartable错误。第一次运行时，它运行正常，但从第二次调用时，我遇到了ReactorNotRestartable。
1.当spider花费的时间超过预期持续时间时，crochet.wait_for()出现超时异常
这篇文章的灵感来自于https://stackoverflow.com/a/57347964/12951298

import sys
import imp
from scrapy.crawler import  CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.utils.log import configure_logging
from twisted.internet import reactor;

from crochet import setup, wait_for

setup()

sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")

@wait_for(900)
def crawl():
    '''
    wait_for(Timeout = inseconds)
    change the timeout accordingly
    this function will raise crochet.TimeoutError if more than 900
    seconds elapse without an answer being received
    '''
    spider_name="header_spider" #your spider name
    project_settings = get_project_settings()
    spider_loader = SpiderLoader(project_settings)

    spider_cls = spider_loader.load(spider_name)
    configure_logging()
    process = CrawlerRunner({**project_settings});
    d = process.crawl(spider_cls);
    return d;

if __name__ == "__main__":
    main('', '')