[BUG] Memory Leak Warning

[ayush-mehta]

Question

I am importing a class Chart() whose object I am creating in my code
Someone raised a github issue with similar issue, there @louisnw01 replied that You must encapsulate the Chart object within an if name == 'main' block.

Now I'll explain what my code is doing
Inside the if name == 'main' block, I am invoking an async function using asyncio.run(main())
Now Inside the main, I am awaiting another async function called processMessage which is written in another file, and inside that processMessage Function I am invoking another function called createChart which is again in another file which is creating the Chart object and it returns the screenshot in bytes array.

I am not sure what I am doing wrong I even tried to delete the chart object which I created from Chart class
This is where I am creating Chart object
I am invoking this function through another function which is called insides if name=='main' block, I am not sure how to fix it

import pandas as pd
from dtos.webhookDtos import SmaCreateChart
import gc

def createChart(data: pd.DataFrame, sma: SmaCreateChart = None):
    chart = Chart()
    if sma:
        line = chart.create_line(name=f'SMA {sma.period}')
        line.set(sma.data)
    chart.set(data)
    chart.show(block=False)
    img = chart.screenshot()
    chart.exit()
    del chart
    gc.collect()
    return img

### Code example

_No response_

[louisnw01]

Hey @ayush-mehta,

Please provide a minimal reproducible example.

Louis

[ayush-mehta]

I have multiple files, I'll start sharing them in this thread for reproducing

[ayush-mehta]

Here is my utils/graphing.py file

import pandas as pd
from dtos.webhookDtos import SmaCreateChart

def createChart(data: pd.DataFrame, sma: SmaCreateChart = None):
    chart = Chart()
    if sma:
        line = chart.create_line(name=f'SMA {sma.period}')
        line.set(sma.data)
    chart.set(data)
    chart.show(block=False)
    img = chart.screenshot()
    chart.exit()
    return img

[ayush-mehta]

Here is my service/webhook.py file

from dtos.webhookDtos import MessageCustomisation, MessageRequest, SmaCreateChart
from utils.computation import calculate_sma, lastLowDiff, todayDiff
from utils.discord import createMessage, send_message_with_image
from utils.graphing import createChart
from utils.data import getData

async def processMessage(requestBody: MessageRequest):
    stocks = requestBody.stocks.split(',')
    prices = requestBody.trigger_prices.split(',')
    TriggerTime = requestBody.triggered_at
    responses = []
    for stock in stocks:
        symbol = stock + ".NS"
        data = await getData(symbol)
        sma = SmaCreateChart(period=20, data=calculate_sma(data, period=20))
        img = createChart(data, sma)
        swingPeriod = 7
        lastLowDiffMsg = MessageCustomisation(text = f'{swingPeriod} days Risk', value = str(round(lastLowDiff(data, period=swingPeriod), 2)))
        todayDiffMsg = MessageCustomisation(text = "Day's Change", value = str(round(todayDiff(data), 2)))
        message = createMessage(symbol, prices[stocks.index(stock)], TriggerTime, custom=[lastLowDiffMsg, todayDiffMsg])
        channelId = requestBody.channelId if requestBody.channelId else CHANNEL_ID
        response = await send_message_with_image(message=message, channel_id=CHANNEL_ID, token=TOKEN, image_bytes=img)
        responses.append(response.json())
    return responses

[ayush-mehta]

And here is my outer most file which I am running, which imports the function from the webhook.py file

from dtos.webhookDtos import MessageRequest
from service.webhook import processMessage
from utils.computation import scanner
from utils.data import insertRowGoogleSheets, loadData, loadSymbols, parseBodyForLogs, updateSymbols
import datetime as dt
import time
import asyncio
import json
from tqdm import tqdm

async def main(timeToSleep: int = 5, numberOfScans: int = 10):
    count = 0
    errors = []
    startTime = time.time()
    
    try:
        symbols = loadSymbols()
    except Exception as e:
        errors.append({"Error in Loading Symbols": str(e)})
    
    try:
        database = loadData()
    except Exception as e:
        errors.append({"Error in Loading Database": str(e)})
    scannedSymbols = ""
    scannedPrices = ""
    scannedTimes = dt.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + ','
    for symbol in tqdm(symbols.get('currSymbols')):
        try:
            scannedResult = await scanner(symbol, database[symbol])
            if scannedResult['valid']:
                scannedSymbols += scannedResult['symbol'].split('.')[0] + ','
                scannedPrices += str(scannedResult['price']) + ','
        except Exception as e:
            errors.append({symbol: str(e)})
    try:
        for symbol in scannedSymbols[:-1].split(','):
            symbols['currSymbols'].remove(symbol + '.NS')
        updateSymbols(database=symbols)
    except Exception as e:
        errors.append({"Error in Updating Symbols": str(e)})
    
    endTime = time.time()
    print(f"Time taken: {endTime - startTime} seconds")
    
    with open('errors.json', 'a') as f:
        json.dump(errors, f, indent=4)
    
    return MessageRequest(stocks=scannedSymbols[:-1], triggered_at=scannedTimes[:-1], trigger_prices=scannedPrices)

if __name__ == "__main__":
    numberOfScans = input("Number of scans: ")
    timeToSleep = input("Time to sleep: ")
    count = 0
    while True:
        requestBody: MessageRequest = asyncio.run(main())
        requestBody.channelId = EQUITY_CHANNEL_ID
        if (requestBody.stocks):
            asyncio.run(processMessage(requestBody=requestBody))
            logs = parseBodyForLogs(requestBody)
            resp : bool = insertRowGoogleSheets(rows=logs)
            if not resp:
                print("Error in inserting logs")
        count += 1
        if count == int(numberOfScans):
            quit()
        asyncio.run(asyncio.sleep(int(timeToSleep)))

[louisnw01]

Hey,

That is not a minimal reproducible example.

Use csv files as shown in the examples; it should be no more than 20-30 lines of code.

Louis

[ayush-mehta]

Hey @louisnw01,
When ever the execution of this file is completed, it shows the following warning

/Users/ayushmehta/anaconda3/envs/trading-bot/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 138 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

After many such code executions and warnings, it gives following error, to fix it I restart my device and then it starts working but with the warnings.
OSError: [Errno 28] No space left on device

I'll go through. your examples and share the minimal reproducible example soon

[ayush-mehta]

Here is the minimal reproducible example @louisnw01 , csv file is not needed as I am fetching data from yfinance, it returns a dataframe. I hope that this helps you reproduce the issue.

import datetime as dt
import yfinance as yf
import pandas as pd
from lightweight_charts import Chart

async def getData(symbol: str, period: int = 200, interval: str = '1d') -> pd.DataFrame:
    today = dt.datetime.today().date() - dt.timedelta(days=period)
    desired_time = dt.time(9, 15)
    chartStartTime = dt.datetime.combine(today, desired_time)
    data = await asyncio.to_thread(yf.download, symbol, start=chartStartTime, interval=interval)
    return data

def createChart(data: pd.DataFrame):
    chart = Chart()
    chart.set(data)
    chart.show(block=False)
    img = chart.screenshot()
    chart.exit()
    return img

async def main():
    data = await getData('TATAMOTORS.NS')
    img = createChart(data)

if __name__ == "__main__":
    asyncio.run(main())

[louisnw01]

Other than the message, I'm not getting any unexpected behaviour

[louisnw01]

The semaphore object message has been on my radar for a while now, but I haven't been able to find a fix. If I do I will push a hotfix, but it doesn't appear to be causing anything strange to happen.

[Jagz003]

I'm also facing the same issue, At first I also got the similar warning but after weeks of daily execution I got OSError: [Errno 28] No space left on device .

[ayush-mehta]

At first it does not cause any strange behaviour, but upon multiple executions, it starts eating space on the device, you can yourself check it, try running the same file for couple of hundreds of time, and you'll see a jump in the memory, and once it reaches above a certain threshold, you'll start getting OSError: [Errno 28] No space left on device that I mentioned earlier.

There is somewhere a memory leak in the service, which I am unable to debug. Have you tried running profilers to find the leak @louisnw01 ?

[bent-verbiage]

@ayush-mehta I had the same issue before, also while taking screenshots of charts in quick succession. While I wasn't able to fully pin down the root cause, it did go away for me. Two things that may have done it:

1. moving to pyview 4.3 (see issue)
2. adding a short sleep after chart.exit()

[ayush-mehta]

Hey @bent-verbiage
I tried working with pywebview==4.3 and I also added time.sleep(1) post chart.exit(), but the warning is persistent.

[bent-verbiage]

Sorry to hear that @ayush-mehta.

I'm no expert but just in case it helps, see the code that I use below. Some of the retry and exception handling is possibly overkill, but once I stopped getting the errors, I left it as it was.

Obviously @louisnw01 is the authority though, and hopefully can find the root cause. And maybe even a way to capture screenshots that include the subcharts :-)

@standard_retry
def take_screenshot(chart, file_name,):
    img = chart.screenshot()
    with open(file_name, 'wb') as f:
        f.write(img)
    
    print(f"Screenshot of {file_name} taken successfully")
    return file_name


async def show_chart_async(chart, file_name, subchart=None, subchart_file_name="", max_retries=3,
                           retry_delay=2):
    attempt = 0
    timeout_duration = 3
    while attempt < max_retries:
        try:
            print(f"Attempting to show and capture chart {file_name}, try {attempt + 1}")
            await asyncio.wait_for(chart.show_async(), timeout=timeout_duration)
            take_screenshot(chart, file_name)
            if subchart:
                take_screenshot(subchart, subchart_file_name)
            chart.exit()
            return
        except asyncio.TimeoutError:
            print(
                f"Timeout occurred: Chart display took longer than {timeout_duration} seconds. Retry after a short delay.")
        except Exception as e:
            print(f"Error displaying or capturing chart on attempt {attempt}:\n {str(e)}")
        await asyncio.sleep(retry_delay)  # Non-blocking sleep
        attempt += 1
    print("All attempts failed to display and capture the chart") 

[ayush-mehta]

@bent-verbiage I tried your approach, I have attached my snippet below, but I am still getting warning of leaked semaphore objects. I even tried commenting out the screenshot segment. Do you get the similar warning message, if not then it could be device or processor specific issue which is bit unlikely. Try running the following snippet and let us know.

I believe the issue is with the chart object itself, the python garbage collector is somehow unable to free up the space. I even tried, to invoke garbage collector using gc.collect() but still it did not work, I even tried to delete the variable using del chart but that also did not fix the issue. I think something inherently issue is with the chart class object. I am going through the source code to understand the codebase and try to come up with a fix.

@louisnw01 it'd be great help if you could share me code documentation which contains LLD of this library. I went through the documentation attached but I believe it is not sufficient for developers to understand and start contributing easily.

import datetime as dt
import yfinance as yf
import pandas as pd
from lightweight_charts import Chart
import time


async def getData(symbol: str, period: int = 200, interval: str = '1d') -> pd.DataFrame:
    today = dt.datetime.today().date() - dt.timedelta(days=period)
    desired_time = dt.time(9, 15)
    chartStartTime = dt.datetime.combine(today, desired_time)
    data = await asyncio.to_thread(yf.download, symbol, start=chartStartTime, interval=interval)
    return data

async def createChart(data: pd.DataFrame):
    chart = Chart()
    chart.set(data)
    await asyncio.wait_for(chart.show_async(block=False), timeout=3)
    img = chart.screenshot()
    chart.exit()
    time.sleep(1)
    return

async def main():
    data = await getData('TATAMOTORS.NS')
    img = await createChart(data)

if __name__ == "__main__":
    asyncio.run(main())

[louisnw01]

@ayush-mehta del chart only deletes the reference, so that probably wouldn't work.

As for LLD, I haven't provided this as of yet but will look to once v2.0 is released.

The code which will be the culprit is located in chart.py; this file contains the main Chart object and other handlers which run pywebview in a seperate process, to allow for non blocking execution.

I do remember this warning did not always show up, so perhaps looking back at older versions until you reach one which doesn't throw the warning might be of use?

[ayush-mehta]

@louisnw01 I'll try going through the previous versions and figure out a cause, this information is really helpful and will ease things up a bit for me.