pandas 执行多个PyODBC sql查询时,Python脚本执行停止

km0tfn4u  于 2023-03-06  发布在  Python
关注(0)|答案(1)|浏览(148)

我试图写一个循环,运行在一个数据库中的多个表上,将结果存储在一个 Dataframe 中,然后将该 Dataframe 写入Excel工作簿。然而,即使我的列表中有29个表,我的代码执行也会在一小段时间后停止(而且不一致,有时它会在1个查询后停止,但通常会在中停止5-6个查询)。有什么想法吗?

conn = pyodbc.connect("DSN=XXXX")
conn.setdecoding(pyodbc.SQL_CHAR, encoding='utf-16-le')
conn.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-16-le')
conn.setencoding(encoding='utf-16-le')
TABLES = ['Table1', 'Table2', ..., 'Table29']

def query_multiple_tables(tables: list[str]) -> list[pd.DataFrame]:
    dataframes = []
    for table in tables:
        QUERY = f"SELECT * FROM [db].[dbo].[{table}]"
        col_crsr = conn.cursor()
        data_crsr = conn.cursor()
        
        cols = [row.column_name for row in col_crsr.columns(table=f"{table}")]
        data = pd.DataFrame.from_records(data_crsr.execute(QUERY).fetchmany(20), columns=cols)
        
        print(f"Extracting table: {table}")
        dataframes.append(data)

        col_crsr.close()
        data_crsr.close()
    
    return dataframes

def write_to_excel(dataframes: list[pd.DataFrame]) -> None:
    
    try:
        with pd.ExcelWriter("Data.xlsx", mode='a', if_sheet_exists='replace') as writer:
            for i, dataframe in enumerate(dataframes):
                dataframe.to_excel(writer, sheet_name=f'{i}')
                print(f"Append successful for: {i}")
    except FileNotFoundError:
        with pd.ExcelWriter("Data.xlsx", mode='w') as writer:
            for i, dataframe in enumerate(dataframes):
                dataframe.to_excel(writer, sheet_name=f'{i}')
                print(f"Initial write successful for: {i}")

data = query_multiple_tables(FIA_TABLES)
write_to_excel(data)
tv6aics1

tv6aics11#

结果将被放入一个df“查询”和循环中,并连接到主df中,然后您可以将其放入csv/xlsx中

import pyodbc
import pandas as pd
connection_string = "DRIVER={SQL SERVER};Server=server;Database=db;Trusted_Connection=yes;"
connection = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection)
connection = engine.connect() 
df = pd.DataFrame()
Tables = ['1', '2'....]
for table in Tables:
   query = f'''
   select * from {table}
'''
   query = pd.read_sql_query(query, connection)
   df = pd.concat(df, query)

相关问题