如何防止在python中读取数据库的线程累积内存使用量?

kgsdhlau  于 2021-06-15  发布在  Mysql
关注(0)|答案(0)|浏览(231)

我有一个代码,它使用一个线程周期性地从mysql数据库中读取数据,并生成该数据的Dataframe。在较短的时间间隔内,它使用一个线程周期性地使用该数据(在本例中是一个简单的打印)。每次读取新数据时,旧数据都变得无关紧要。问题是,每次第一个线程(获取数据)运行时,此代码都会累积内存使用量。如何防止这种情况?
我尝试过垃圾收集器,但没有成功。


## import libraries

from time import sleep
import mysql.connector as sql
import pandas as pd
import threading
import numpy as np
import gc

## define function to get data from mysql database and make pandas dataframe

def get_HLC():
    db=sql.connect(host='localhost',user='root',password='',database='algo')
    df=pd.read_sql('select *from ticks',con=db,parse_dates=True)
    df=pd.DataFrame(df)
    df=df.set_index(['timestamp'])
    df.index = pd.to_datetime(df.index.astype(np.int64), unit='ms')
    df2 = df.resample('60s', how={'last_price': 'ohlc'})
    df3 = df.resample('60s', how={'volume': 'sum'})
    df3.columns = pd.MultiIndex.from_tuples([('volume', 'sum')])
    df4 = pd.concat([df2, df3], axis=1)
    df4.iloc[:,3] = df4.iloc[:,3].fillna(method='ffill')
    df4.iloc[:,0] = df4.iloc[:,0].fillna(value=df4.iloc[:,3])
    df4.iloc[:,1] = df4.iloc[:,1].fillna(value=df4.iloc[:,3])
    df4.iloc[:,2] = df4.iloc[:,2].fillna(value=df4.iloc[:,3])
    df5 = df4.iloc[-230:,:]
    db.close()
    del(db, df, df2, df3, df4)
    gc.collect()
    return df5['last_price']['high'], df5['last_price']['low'], df5['last_price']['close']

## define function that gets the data in a loop

def thread1_Function():
    while True:
        global low
        df5 = get_HLC()
        low = df5[1]
        del(df5)
        gc.collect()
        sleep(30)

## start a thread of the get data loop

thread1 = threading.Thread(target=thread1_Function)
thread1.start()

## pause

sleep(5)

## define function that loops and does things with data of thread1

def thread2_Function():
    while True:
        #do things with thread1 data
        print(low)
        sleep(15)

## start a thread of the do things with data of thread1 loop

thread2 = threading.Thread(target=thread2_Function)
thread2.start()

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题