json 在两个不同的SQL服务器之间进行交叉引用?

aemubtdh  于 2023-10-21  发布在  其他
关注(0)|答案(1)|浏览(88)

我重新发布这个,因为我的最后一个问题令人困惑。下面是我的代码:

import pyodbc
import logging
import json

def read_query():
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)

    with open(r'test_scripts\upsert_config.json','r') as ts:
        config = json.load(ts)

    driver = config['config_info']['driver']
    source_server = config['config_info']['source_server']
    target_server = config['config_info']['target_server']
    source_database = config['config_info']['source_database']
    target_database = config['config_info']['target_database']

    try:
        conn = pyodbc.connect(
        f'Driver={driver};'
        f'Server={source_server};'
        f'Server={target_server};'
        f'Database={source_database};'
        f'Trusted_Connection=yes;'
        )

        logger.info("Successfully connected to database")

    except Exception as e:
        logger.error("Unable to connect to database: %s", str(e))

    try:
        target_database=f'{target_database}.dbo'
        source_database=f'{source_database}.dbo'
        cursor = conn.cursor()
        read_table_query="""
            SELECT TOP 3 table_name
            FROM information_schema.tables
            ORDER BY table_name asc;
            """
        cursor.execute(read_table_query)         

        for tables in cursor.fetchall():
            tab = tables[0]
            select_data_query = f'INSERT INTO {target_database}.{tab} select * FROM {source_database}.{tab} where PRCS_DTE > DATEADD(day, -3, CONVERT (date, SYSDATETIME()));'
            cursor.execute(select_data_query)
            print("Query ran successfully, {} records inserted ".format(cursor.rowcount) + f"into {tab}")
            conn.commit()
            

    except Exception as e:
        logging.exception(e)

read_query()

下面是我的JSON配置文件:

{
    "config_info":
    {
        "driver": "ODBC Driver 17 for SQL Server",
        "source_server": "source_sql_server_name",
        "target_server": "target_sql_server_name",
        "source_database": "source_sql_db_name",
        "target_database": "target_sql_db_name"
    }
    
}

我不能从我的源数据库中选择数据并插入到我的目标数据库中,因为它们位于两个不同的sql服务器上。我认为最好的方法是拆分我的查询,因此我从1个数据库服务器中选择数据,然后将该数据插入目标数据库服务器。我试着将结果写入pandas的框架中,并添加了以下内容:df=pd.read_sql(sa.text(select_data_query), conn, chunksize=10000)但它说The first argument to execute mu st be a string or unicode query.
我被卡住了,我不知道如何简单地做到这一点
更新代码:
我修改了我的代码,现在看起来像这样:

try:
        source_database=f'{source_database}.dbo'
        cursor = conn.cursor()
        read_table_query="""
            SELECT TOP 10 table_name
            FROM information_schema.tables
            ORDER BY table_name asc;
            """
        cursor.execute(read_table_query)         

        for tables in cursor.fetchall():
            tab = tables[0]
            select_data_query = f'select * FROM {source_database}.{tab} where PRCS_DTE > DATEADD(day, -3000, CONVERT (date, SYSDATETIME()));'
            df=pd.read_sql(select_data_query, conn, chunksize=10000)
            cursor.execute(select_data_query)
            print("Query ran successfully, {} records selected ".format(cursor.rowcount) + f"from {tab}")

        engine = sa.create_engine(f'mssql+pyodbc://@{target_server}/{target_database}?trusted_connection=yes&driver={target_driver}')
   
        for chunk_dataframe in df:
            chunk_dataframe.to_sql(f'{tab}', engine, if_exists='append', index=False, method='multi')
            print("Query ran successfully, {} records inserted ".format(cursor.rowcount) + f"into {tab}")

    except Exception as e:
        logging.exception(e)

read_query()

它运行正常,直到它到达一个有数据的表。然后它给出了错误:
pyodbc.Error:(“HY000”,“[HY000] [Microsoft][ODBC Driver 17 for SQL Server]连接因另一个命令(0)的结果而忙碌(SQLExecDirectW”)
我该怎么修复它?

pb3s4cty

pb3s4cty1#

从第一个代码示例开始尝试:

cursor.execute(select_data_query)
cursor.execute("commit;") 
print("Query....

相关问题