pandas > to_sql > check_case_sensitive中sys.tables上的SQL死锁

soat7uwm  于 2023-08-01  发布在  其他
关注(0)|答案(1)|浏览(94)

我遇到了一个问题,我在Azure SQL数据库上看到死锁。这是因为我们将代码从虚拟机更改为Airflow。这意味着最明显的变化是:

  • Windows Server 2022 Datacenter Azure Edition到Kubernetes 1.25.6
  • SQLAlchemy和/或pandas的版本更改。

df.to_sql中,它总是一个if_exists='replace',然后它总是在check_case_sensitive上崩溃,在sys.tables上死锁。如果它是在一个实际的数据表上,我会理解这是一个类似于事务提交的问题,如果我自己检查sys.tables,我可能会理解它,但错误是在模块内部提出的。大小写检查只是一种检查,以引发警告“在写入表后,在数据库中找不到所提供的表名“{name}”,可能是由于区分大小写的问题。考虑使用小写的表名。”甚至没有修复它。

select [INFORMATION_SCHEMA].[TABLES].[TABLE_NAME]
from [INFORMATION_SCHEMA].[TABLES]
where [INFORMATION_SCHEMA].[TABLES].[TABLE_SCHEMA] = CAST('dbo' as nvarchar(max))
    and [INFORMATION_SCHEMA].[TABLES].[TABLE_TYPE] = CAST('BASE TABLE' as nvarchar(max))
order by [INFORMATION_SCHEMA].[TABLES].[TABLE_NAME]

字符串
SQL DB上没有完整的事务日志。我有什么可能性才能找到问题呢?我想不出办法了。
回溯:

Traceback (most recent call last):
  File "/opt/airflow/dags/repo/dags/xyz.py", line 308, in xyzabc
    df.to_sql(tablename, con=engine, if_exists=if_exists, dtype=datatypedict, index=False, chunksize=10000)
  File "/home/airflow/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 2878, in to_sql
    return sql.to_sql(
  File "/home/airflow/.local/lib/python3.10/site-packages/pandas/io/sql.py", line 769, in to_sql
    return pandas_sql.to_sql(
  File "/home/airflow/.local/lib/python3.10/site-packages/pandas/io/sql.py", line 1932, in to_sql
    self.check_case_sensitive(name=name, schema=schema)
  File "/home/airflow/.local/lib/python3.10/site-packages/pandas/io/sql.py", line 1832, in check_case_sensitive
    table_names = insp.get_table_names(schema=schema or self.meta.schema)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 266, in get_table_names
    return self.dialect.get_table_names(
  File "<string>", line 2, in get_table_names
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
    ret = fn(self, con, *args, **kw)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/dialects/mssql/base.py", line 2613, in wrap
    return _switch_db(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/dialects/mssql/base.py", line 2655, in _switch_db
    return fn(*arg, **kw)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/dialects/mssql/base.py", line 3065, in get_table_names
    table_names = [r[0] for r in connection.execute(s)]
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1948, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2129, in _handle_dbapi_exception
    util.raise_(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('40001', '[40001] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Transaction (Process ID 418) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. (1205) (SQLExecDirectW)')
[SQL: SELECT [INFORMATION_SCHEMA].[TABLES].[TABLE_NAME] 
FROM [INFORMATION_SCHEMA].[TABLES] 
WHERE [INFORMATION_SCHEMA].[TABLES].[TABLE_SCHEMA] = CAST(? AS NVARCHAR(max)) AND [INFORMATION_SCHEMA].[TABLES].[TABLE_TYPE] = CAST(? AS NVARCHAR(max)) ORDER BY [INFORMATION_SCHEMA].[TABLES].[TABLE_NAME]]
[parameters: ('dbo', 'BASE TABLE')]
(Background on this error at: https://sqlalche.me/e/14/dbapi)

v1uwarro

v1uwarro1#

正如Martin Smith在评论中指出的:
在Azure中,您可以通过查询主数据库来获取死锁图,如这里的Lesson Learned #19: How to obtain the deadlocks of your Azure SQL Database or Managed Instance?。你应该那样做看看确切的原因
这样做后,我们发现它在同一数据库上工作的完全不同的进程上死锁,无论是否在Airflow上运行,我们之前可能没有注意到。它不会在自身、前一个查询和/或多线程上死锁。
以下几种选择:

  • 存在与迁移无关的外部原因(例如:数据库池忙碌)
  • 在迁移过程中,我们更多地关注阅读日志,从而发现它以前看不到的地方
  • Airflow有一个更精确的启动时间,在Windows上,如果你一次启动10个进程,进程会被推回,导致Airflow上更多的重叠。

我们还没有找到确切的原因,但我相当肯定这不是一个特定的Pandas问题。另一个人的答案是:检查死锁,它可能不是你想的那样。它很可能不是SQLAlchemy或Pandas死锁本身。在Azure弹性池上,检查池使用情况。在SQL Server上,检查您的资源。
您不需要Azure SQL Auditing或其他可选的事务式监视,它是一个非常基本的SQL Server系统功能。

相关问题