使用SQLAlchemy和Scrapy导出到SQLite时出错

csga3l58  于 2022-11-23  发布在  SQLite
关注(0)|答案(1)|浏览(120)

我正在创建一个从etf.com收集etf数据的scraping bot。我试图将收集的数据导出到sqlite数据库,但每次这样做时,我都会收到以下错误消息(对于我尝试添加的每个项目):

sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 0 - probably unsupported type.
[SQL: INSERT INTO etf (ticker, name, issuer, aum, expense_ratio, tr, segment) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: (['ROKT'], ['SPDR S&P Kensho Final Frontiers ETF'], ['State Street Global Advisors'], ['$19.24M'], ['0.45%'], ['6.45%'], ['Equity: U.S. Space'])]

我的刮刀(brandetfs_spider.py):

for etf in etfs:
                    loader = ItemLoader(item=BrandetfsItem(), selector=etf)
                    loader.add_css('ticker', 'a.linkTickerName::text')
                    loader.add_css('name', 'td.col_2::text')
                    loader.add_css('issuer', 'td.col_3::text')
                    loader.add_css('aum', 'td.col_4::text')
                    loader.add_css('expense_ratio', 'td.col_5::text')
                    loader.add_css('tr', 'td.col_6::text')
                    loader.add_css('segment', 'td.col_7::text')

                    yield loader.load_item()

型号:

Base = declarative_base()

def db_connect():
    """
    performs database connection using database settings from settings.py
    returns sqlalchemy engine instance
    """
    return create_engine(get_project_settings().get("CONNECTION_STRING")) # connects to a database

def create_table(engine):
    Base.metadata.create_all(engine)

class ETF(Base):
    __tablename__ = "etf"
    print("-------------------------------------------")

    id = Column(Integer, primary_key=True)
    ticker = Column('ticker', Text())
    name = Column('name', Text())
    issuer = Column('issuer', Text())
    aum = Column('aum', String(10))
    expense_ratio = Column('expense_ratio', Text())
    tr = Column('tr', Text())
    segment = Column('segment', Text())

管线:

class BrandetfsPipeline(object):
    def __init__(self):
        """
        Initializes database connection and sessionmaker
        creates tables
        """
        engine = db_connect()
        create_table(engine)
        self.Session = sessionmaker(bind=engine)

    def process_item(self, item, spider):
        """
        Save etfs in the database
        This method is called for every item pipeline component
        """
        session = self.Session()
        # create etf table
        etf = ETF()
        etf.ticker = item["ticker"]
        etf.name = item["name"]
        etf.issuer = item["issuer"]
        etf.aum = item["aum"]
        etf.expense_ratio = item["expense_ratio"]
        etf.tr = item["tr"]
        etf.segment = item["segment"]

        try:
            session.add(etf)
            session.commit()
        except:
            session.rollback()
            raise
        finally:
            session.close()

        return item

settings.py:

CONNECTION_STRING = 'sqlite:///scrapy_etfs.db'

我不明白为什么我会收到错误消息,因为我正在为数据库字段使用文本类型,而这正是我试图放入它们的内容

vhipe2zx

vhipe2zx1#

这可能是因为itemloader返回列表。您可以通过使用TakeFirst作为处理器来修复它

相关问题