Garbage Collection - Python.Runtime.Dll throws System.AccessViolation

[AgazzottiR]

Environment

• Pythonnet version: 3.0.2
• Python version: 3.9
• Operating System: Windows 10
• .NET Runtime: .NET Framework 4.8

Details

I am trying to open a large file through a .NET Framework DLL and access the data inside python.

Scenario:

• Pythonnet is loaded inside the module in which I perform these operations.
• The C# Object is an attribute of a Python class.
• The python class has the __del__() method implemented in which I have to call a method, inside the C# class, right before destructing the object.
• I have added a log print in the C# Destructor to know when the CLR calls it.

Errors

• In some cases, the CLR logs the destructor call before the __del__() of the python method is executed, so this is not a big deal because the method I have to call C# side is also called in the destructor, and I mange with exception handling the fact that the C# object is no longer there when the __del__() is executed. Solvable But Not Nice.
• In other cases, the python GC frees the memory before the Pythonnet.Runtime.Dll manages to deal with it, so in the function Python.Shutdow() (Where you try to get pointers if you are familiar with the code) you are trying to get rid of already freed memory areas( Sorry if it is a rough explanation but I do not have time and resources to do a full reverse engineering of this problem). I have solved it by adding try and catches here and there inside Python.Shoutdown() function and by checking null pointers. VERY BAD

If someone with more experience than me about this tool sees any issues in what I am trying to do python side I will happily fix my code, otherwise if you think this might be a real problem I will do a pull request with my corrections so everyone can get a benefit from them.

UPDATE ADDED CODE

• Python Class Code:

import pythonnet
from pythonnet import load
load("netfx")
import clr
from time import time
try:
    clr.AddReference('FileManager.Library')
    import FileManager.Library
except System.IO.FileNotFoundException:
    print("[File loader] DLL not found!")

class FileLoader:
    def __init__(self, filePath):
        self.dataLoader = FileManager.Library.FileReader(filePath)
        self.isClosed = False

    def load_data(self):
        return self.dataLoader.GetDataPoints()
    
    def close(self):
        self.dataLoader.Close()
        self.isClosed = True

    def __del__(self):
        print(f"[FileLoader - {time()}] Freeing memory!", flush=True)

        if not self.isClosed:
            self.close()

• Python Main:

from FileManager.FileLoader import FileLoader


if __name__ == "__main__":
    fl = FileLoader("data.csv")
    data = fl.load_data()
    print("Done!")

• C# ClassLibrary Code:

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace FileManager.Library
{
    public class FileReader: IDisposable
    {
        private string filePath;
        private int objectId;
        private bool isDisposed = false;
        private List<DataPoint> fileContent;
        private DateTimeOffset dto = new DateTimeOffset(DateTime.UtcNow);

        ~FileReader() {
            if(!isDisposed)
                Dispose();

            System.Console.WriteLine($"[C# -  {dto.ToUnixTimeMilliseconds().ToString()} ] Deleting {objectId}");
        }

        public FileReader(string filePath)
        {
            this.filePath = filePath;
            fileContent = new List<DataPoint>();
            objectId = this.GetHashCode();
            LoadFile();
        }

        private void LoadFile()
        {

            if (!File.Exists(filePath))
            {
                throw new FileNotFoundException($"Cannot open file{filePath}");
            }

            const Int32 BufferSize = 128;
            using (var fileStream = File.OpenRead(filePath))
            using (var streamReader = new StreamReader(fileStream, Encoding.UTF8, true, BufferSize))
            {
                String line;
                while ((line = streamReader.ReadLine()) != null)
                {
                    var data = line.Split(',');
                    var dataPoint = new DataPoint((int)Convert.ToDouble(data[0]), Convert.ToDouble(data[1]));
                    fileContent.Add(dataPoint);
                }
                fileStream.Close();
            }
            
            return;
        }

        public DataPoint GetDataPoint(int i)
        {
            return fileContent[i];
        }

        public List<DataPoint> GetDataPoints()
        {
            return fileContent;
        }

        public void Dispose() {
            
            if (isDisposed) return;
            System.Console.WriteLine($"[C# - {dto.ToUnixTimeMilliseconds().ToString()}] Disoposing Obj {objectId}");
            isDisposed = true;
        }

        public void Close() {
            System.Console.WriteLine("Dummy things That I have to do before closing");
        }

    }
}

Running the python main above with the debugger works fine(also time stamps in correct order) but without the debugger I ALWAYS get the following error:

Dummy things That I have to do before closing
Done!
[C# - 1698683227427] Disoposing Obj 26765710
[C# -  1698683227427 ] Deleting 26765710
[FileLoader - 1698683227.5009353] Freeing memory!
Exception ignored in: <function FileLoader.__del__ at 0x00000158F00920D0>
Traceback (most recent call last):
  File "FileLoader.py", line 30, in __del__
TypeError: 'MethodObject' object is not callable

**Where am I wrong? How Can I fix it? **
Note 1: Obviously I do not want to try catch it I want it to work properly
Note 2: Same main in C# no errors at all

[filmor]

Can you please provide a bit of code that shows this behaviour (or better yet, a reproducible example)? If everything is working correctly, C# objects should never be finalised while they are still referenced. Also, did you ensure that you log statements are flushed immediately and/or are timestamp at a very high resolution? Since you are dealing here with potentially different buffers, it can easily happen that the logging order is off, leading you down a wrong path.

[AgazzottiR]

I cannot provide an example immediately but if it is strictly necessary I can write one. But what I can tell for sure is that when I jump in to the __del__() function and I have already seen the log prints, any method on the C# object, that I was able to call for the whole previous lifetime of the object, throws python side "Attribute error with MethodObject is not callable"....so from this I am deducing that the destructor logs are right....otherwise I cannot explain to myself why any method that has always worked in the previous object lifetime is suddenly no longer callable...