pymysql.err.operationalerror:2013,使用pymysql和sqlalchemy

b4wnujal  于 2021-06-24  发布在  Mysql
关注(0)|答案(2)|浏览(828)

伙计们!希望有人能帮我解决这个问题。
我通过sqlalchemy执行一个查询,返回需要在python脚本上处理的~6kk行(它是历史数据)。我有一些函数可以使用dataframe读取和处理数据。以下是函数:

def consulta_db_cancelamentos(db_con, query):
    engine = create_engine(db_con, pool_recycle=3600)
    con = engine.connect()
    query_result = con.execution_options(stream_results=True).execute(query)
    query_result_list = []
    while True:
        rows = query_result.fetchmany(10000)
        if not rows:
            break
        for row in rows:
            data = row['data'],\
                row['plano'],\
                row['usuario_id'],\
                row['timestamp_cancelamentos'],\
                row['timestamp'],\
                row['status']

            query_result_list.append(data)

        df = pd.DataFrame()
        if df.empty:
            df = pd.DataFrame(query_result_list)
        else:
            df.append(pd.DataFrame(query_result_list))
        df_cor = corrige_cancelamentos_df(df, '2017-01-01', '2017-12-15')

    con.close()

    return df_cor

如您所见,我已经在尝试读取数据并将其处理/存储在10k行中。当我尝试执行整个脚本时,我在函数中遇到了这个错误(我还包括在 main() ):

Traceback (most recent call last):
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
    l = self.process_rows(self._fetchmany_impl(size))
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
    row = self._fetchone_impl()
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
    self.__buffer_rows()
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
    self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
  File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
    row = self.read_next()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
    return self._conv_row(self._result._read_rowdata_packet_unbuffered())
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
    packet = self.connection._read_packet()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
    recv_data = self._read_bytes(bytes_to_read)
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aiquis/EI/cancelamentos_testes5.py", line 180, in <module>
    main()
  File "/home/aiquis/EI/cancelamentos_testes5.py", line 164, in main
    cancelamentos_df_corrigido = consulta_db_cancelamentos(db_param, query_cancelamentos)
  File "/home/aiquis/EI/cancelamentos_testes5.py", line 14, in consulta_db_cancelamentos
    rows = query_result.fetchmany(1000)
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1166, in fetchmany
    self.cursor, self.context)
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
    l = self.process_rows(self._fetchmany_impl(size))
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
    row = self._fetchone_impl()
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
    self.__buffer_rows()
  File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
    self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
  File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
    row = self.read_next()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
    return self._conv_row(self._result._read_rowdata_packet_unbuffered())
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
    packet = self.connection._read_packet()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
    recv_data = self._read_bytes(bytes_to_read)
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
Exception ignored in: <bound method MySQLResult.__del__ of <pymysql.connections.MySQLResult object at 0x7f8c543dc198>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1345, in __del__
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1447, in _finish_unbuffered_query
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 991, in _read_packet
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1022, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'
[Finished in 602.4s]

我写的方式 consulta_db_cancelamentos 已经是对so和sqlalchemy文档进行搜索的结果。假设我无法访问mysql服务器管理。
当我限制我的查询只为一个 usuario_id 例如(大约50行)它工作正常。我在mysql工作台上执行了相同的查询,duration/fetch为251.998秒/357.541秒

mnowg1ta

mnowg1ta1#

我面临这样的问题。我把它放进去之后就遇到了问题 stream_results=True 到查询。
对我来说,参数 net_write_timeout 是在制造麻烦。什么时候 stream_results=True 而且应用程序不会立即读取从mysql服务器发送的所有数据,然后在mysql服务器端,到应用程序的写入通信(发送数据包)被阻塞。 net_write_timeout 似乎是一个参数,用于控制mysql服务器写入被阻止的套接字操作所允许的连接秒数。
所以,我变了 net_write_timeout 参数来自 60 (默认)到 3600 解决了问题。

wgeznvg7

wgeznvg72#

已解决在mysql server中执行此命令的问题:

set global max_allowed_packet = 67108864;

此解决方案建议在查询过程中与mysql服务器断开连接

相关问题