Memory leaks - After running 24 hours it crashes

[mgungorchamp]

Description

I have a very simple APP - which shows web pages and images in turn. I am using MS VS Profiler and Objects and Heapisize gradually increases and at the end OOM - Out of Memory happens and the app crashes.
I thought it was on my part, I was using Singleton, and I removed all singleton, no help. Second I was using a recursive call, I went to use while(true), but no help. I was using SemaphoreSlim, and removed it, no help. I took all the blame, but none was helping me.

I though it was Grid view, though Grid view causes a faster OOM crash, but removing it also did not help

The profiler does not show anything about my code doing wrong... This app should run all the time on screens, but it's not usable, I am planning to move from Net MAUI to Native Android Development. I was hoping to get one stone for 4 birds, but I have no bird at all.

I am not sure if anybody will hear my voice... I feel I wasted my months on this app.

Steps to Reproduce

1- Create an app with an Image and Webview on the content page -
2- Then just show images in rotation and websites, Wait ~24 hours on the Android Device will crash, but Windows will survive since there are vast amount of memory
3-Open profiler and observe Objects and Heapisize grow gradually until it can no longer get tolerated by the OS

Link to public reproduction project repository

https://github.com/mgungorchamp/InfoBoard/tree/Semaphore-NoSingletonNoRecursive/InfoBoard

Version with bug

7.0.92

Is this a regression from previous behavior?

No, this is something new, Not sure, did not test other versions

Last version that worked well

Unknown/Other

Affected platforms

Android, Windows

Affected platform versions

All Android

Did you find any workaround?

No workaround

Relevant log output

At the end you will see OOM message and crash.
[cr_ChildProcessConn] onServiceDisconnected (crash or killed by oom): pid=10851 bindings:W  S

[akhanalcs]

This is wild. I thought they fixed all memory leaks, but apparently not. I hope they look at this soon and get your app working.

[kcrg]

@affableashish I think the vast majority of memory leak issues have only been resolved in .NET 8 branches.

[samhouts]

Yeah, we've put most of the memory leak fixes in .NET 8 and not in .NET 7. Before you give up, you might want to test with the latest previews to see if it improves. If it doesn't, let us know! We're still working diligently on tracking these down. Thanks!!!!

[ghost]

Hi @mgungorchamp. We have added the "s/try-latest-version" label to this issue, which indicates that we'd like you to try and reproduce this issue on the latest available public version. This can happen because we think that this issue was fixed in a version that has just been released, or the information provided by you indicates that you might be working with an older version.

You can install the latest version by installing the latest Visual Studio (Preview) with the .NET MAUI workload installed. If the issue still persists, please let us know with any additional details and ideally a reproduction project provided through a GitHub repository.

This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[mgungorchamp]

I will test with the new version of NET MAUI, for now I wanted to put this here.
More information on this:

debug-console-crash-report.txt

[ShiftyTR]

My application crashes after running for about 4 to 5 hours. However, the console app, which uses the same library, works without any issues. I attempted to use .NET 8 to check for memory leaks, but in release mode, it gets stuck on loading.
I started a new project from scratch and tried to build it in release mode, but it gets stuck during loading. Later, when I initiated a standard Maui project and installed the SignalR client, it keeps connecting and disconnecting continuously. It seems that SignalR has some issues when operating in release mode on .NET 8. At this point, I'm at a loss and have exhausted all potential solutions. I've decided to switch to Java and develop a native application. Unfortunately, this means that my 5 to 6 month-long project has gone to waste.

[mgungorchamp]

Yeah, we've put most of the memory leak fixes in .NET 8 and not in .NET 7. Before you give up, you might want to test with the latest previews to see if it improves. If it doesn't, let us know! We're still working diligently on tracking these down. Thanks!!!!

I am trying with the .NET 8. The profiler is not available in the PREV, and debugging to the Android connection gets lost somehow, therefore I do not see when it crashed and why. I opened with log cat.

I see these two messages... Seems NET 8 is not there yet. If I am doing something wrong I am open... but it looks I am exhausted the options.
Time Device Name Type PID Tag Message
08-31 01:12:48.099 Amazon AFTKA Error 6411 libc Fatal signal 7 (SIGBUS), code 2 (BUS_ADRERR), fault addr 0xb0a45456 in tid 6411 (board.infoboard), pid 6411 (board.infoboard)

Time Device Name Type PID Tag Message
08-31 12:31:46.842 Amazon AFTKA Error 24328 libc Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 24719 (RenderThread), pid 24328 (board.infoboard)

And from debug
[Choreographer] Skipped 298 frames! The application may be doing too much work on its main thread.

I am clueless ...

[samhouts]

@jonathanpeppers any recommendations?

[jonathanpeppers]

The app above (Semaphore-NoSingletonNoRecursive branch) runs for me using a local build of dotnet/maui/main:

@mgungorchamp what steps do I take to see the problem? Do I need to login?

This page probably has a lot of info on how to troubleshoot issues like this:

• https://github.com/dotnet/maui/wiki/Memory-Leaks

But if I can just run your app and see for myself, I'm happy to investigate. Thanks!

[mgungorchamp]

To get the image and URL to show you need to add them via the https://infopanel.vermontic.com/
It's free to create an account and add any web content or upload images then it will start showing them in an ordered fashion.
I have been here at https://github.com/dotnet/maui/wiki/Memory-Leaks and tried to apply most of the stuff, but I could not solve the problem...

[mgungorchamp]

Could you try this branch which has the latest NET 8
https://github.com/mgungorchamp/InfoBoard/tree/2023-08-31-IndPageByCodeNoGrid/InfoBoard
It works but after some hours it crashes... You need to keep the app running... and it will stop sometimes while you are not paying attention.

I use FireTV Stick to test and also Chromecast... it works on tablets/PCs longer time since they have so much memory.

[jonathanpeppers]

Which do you think is the problem <WebView/> or <Image/>? I don't see how replacing the source on these controls would leak, is that basically all the app does? Can you reproduce this in a smaller sample with a single page?

One thought looking at this code: can you clear the WebView's source when it is not visible?

I keep getting this popup, so I'm not sure if I'm able to see the issue or not:

[mgungorchamp]

This is a caught exception, I could not find the solution for it, but seems harmless - due to socket communication - you can ignore it. I tested it on Image and it worked for several days. The image is ok. Webview with Grid faster crash. That's why I tried to tie page Content to webview to eliminate any middleman.

By clearing you mean this webView.Source = null; I did not but did not see the behavior change.

protected override void OnNavigatedFrom(NavigatedFromEventArgs args)
{
base.OnNavigatedFrom(args);
webView.Source = null;
}

For your question: Can you reproduce this in a smaller sample with a single page?
I am testing just that case.
In NET 8 PREV, the profiler is not available therefore I am unable to monitor the object count and heap size changes.

[jonathanpeppers]

In NET 8 PREV, the profiler is not available

What does this mean? Basically, every day I use dotnet-trace and dotnet-dsrouter to profile .NET 8 Android apps without issue.

For your question: Can you reproduce this in a smaller sample with a single page?
I am testing just that case.

An app with only one page with one control, for example, would be great. Right now, there is a lot going on in your app, so it is difficult to pin down where the exact problem is.

[mgungorchamp]

VS 2022 has this option Debug / Windows / Show Diagnostic Tools
But PREV does not have - I will try to use the tools you have mentioned

I will try to do that, but testing could take more than a day since it crashes after a day. Currently, with image / and webview, it crashes less than a day.

[jonathanpeppers]

Should I be looking at this issue running on Android or Windows? I don't believe the Diagnostic Tools have ever worked for mobile, but we'd like them to one day.

[mgungorchamp]

Android!!! Please

[jonathanpeppers]

Many places you are converting JSON web responses to string, such as:

https://github.com/mgungorchamp/InfoBoard/blob/c0cacffa91721ea79eaf40d3dfe442160fe8e6c6/InfoBoard/Services/RestService.cs#L290-L291

This is creating a string in memory the size of every JSON response. Can you use a Stream directly to avoid this?

Example here:

• http://jonathanpeppers.com/Blog/improving-http-performance-in-xamarin-applications

Are you able to reproduce the issue without any HttpClient usage involved?

I have not seen the app crash yet on my Pixel 5, I took a diff several minutes apart and there are is only 90kb more memory used:

I suspect this is not showing a leak for me.

Do I need to setup many images on the site? (https://infopanel.vermontic.com/) Or will a single one show the issue?

[jonathanpeppers]

I have it cycling between two images now and the memory goes up and down:

Now, keep in mind, I am basically using .NET 8 (dotnet/maui/main), so maybe we have resolved the issue?

This one comes to mind, which actually fixed a leak in Image on all platforms:

• [ios] fix memory leak in Image #15062

[jonathanpeppers]

So, I think we're we are at, is to try to make a smaller sample project that has the same problem.

Unfortunately, I do not have an arm device, only arm64 and x64 devices.

[ghost]

Hi @mgungorchamp. We have added the "s/needs-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[mgungorchamp]

Thank you for your diagnosis. I am not sure what other surprises waiting for me down the road. I spent so much time on this. I am moving to Android Studio and Java. Thanks for the help.

[akhanalcs]

C'mon @mgungorchamp, don't give up on MAUI just yet.

[mgungorchamp]

Thank you for the kind word @affableashish but I am not sure when they are going to fix this memory leak - no timeline - this app should work days without restart... I don't want to move away after this many months of learning investment...

[akhanalcs]

I completely understand you @mgungorchamp.
Maybe @jonathanpeppers will provide some further guidance here.

In the meantime, can you implement a workaround?
The workaround I can think of is:
Write a very simple console app or worker service that runs 24hrs in the background that will restart your MAUI app everyday at midnight or something. This way your app will run forever with periodic restarts.

After MAUI team puts a reliable solution, you can get rid of that worker service.
I can probably get you an example of how to do this for windows apps tomorrow because I've done this kind of thing before. Haven't done this for android apps but there's probably a way.

Also please reopen this issue. That way, this issue can make it to MAUI team's backlog and will get worked on (hopefully soon).

[mgungorchamp]

Thanks again @affableashish for the message. But this app will be distributed via the app store, ordinary users cannot do this separate procedure. If there is a way to implement that feature, like when the app crashes, it restarts itself! I don't think this is logically possible, since the app will not be aware of its death. I have started a parallel project, but if this gets solved sooner, I will go back to MAUI before I go so far with Android Studio.

[akhanalcs]

Not sure how much it's worth it but you can create a simple watchdog application and deploy it alongside your app.
I asked ChatGPT so it might not be all correct, but it's a start.

References: https://medium.com/@deepakgahlot98/a-never-ending-service-android-263ec5d5dc3b

ChatGPT answer:

Step 1: Create a Watchdog Android App

Define Permissions:
In your AndroidManifest.xml, define the necessary permissions. Since your watchdog app needs to restart other apps, you should include the RECEIVE_BOOT_COMPLETED permission for automatic startup.

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />

Create a Service:
Create a background service that will periodically check and restart your target app.

public class WatchdogService extends Service {

    private static final long RESTART_INTERVAL = 24 * 60 * 60 * 1000; // 24 hours in milliseconds

    @Override
    public int onStartCommand(Intent intent, int flags, int startId) {
        // Create a timer to periodically restart the target app
        Timer timer = new Timer();
        timer.schedule(new TimerTask() {
            @Override
            public void run() {
                restartTargetApp();
            }
        }, 0, RESTART_INTERVAL);

        return START_STICKY;
    }

    private void restartTargetApp() {
        // Code to restart your target app (the one with the defect)
        try {
            Intent launchIntent = getPackageManager().getLaunchIntentForPackage("com.example.targetapp");
            if (launchIntent != null) {
                startActivity(launchIntent);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Nullable
    @Override
    public IBinder onBind(Intent intent) {
        return null;
    }
}

Add Receiver in Manifest:
Register your service to receive the BOOT_COMPLETED broadcast, so it starts automatically when the device boots up.

<receiver
    android:name=".BootReceiver"
    android:enabled="true"
    android:exported="true">
    <intent-filter>
        <action android:name="android.intent.action.BOOT_COMPLETED" />
    </intent-filter>
</receiver>

Create a BootReceiver:
Create a broadcast receiver to start your watchdog service when the device boots up.

public class BootReceiver extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        if (intent.getAction() != null && intent.getAction().equals(Intent.ACTION_BOOT_COMPLETED)) {
            Intent serviceIntent = new Intent(context, WatchdogService.class);
            context.startService(serviceIntent);
        }
    }
}

Step 2: Deploy Your Watchdog App

Build Your Watchdog App:
Build your watchdog app using Android Studio. Make sure you have the APK file ready.

Install the Watchdog App:
You can install the watchdog app on your device using ADB or by uploading it to a file-sharing service and downloading it on your device. Alternatively, you can publish it on the Google Play Store if necessary.

Step 3: Configure Your Watchdog App

Configure Package Name:
In the restartTargetApp method of your WatchdogService, replace "com.example.targetapp" with the package name of your crashing app.

Set Restart Interval:
Adjust the RESTART_INTERVAL to your preferred duration for restarting the target app.

Step 4: Run Your Watchdog App
Start your watchdog app. It will automatically start monitoring and periodically restart your target app.

Ensure that your target app is also installed on the same device.

[akhanalcs]

I agree it's one hell of a workaround and would be nice when MAUI/ runtime team fixes this.

[jonathanpeppers]

Maybe just to recap where we are on this issue:

1. I did not observe any memory leaks using dotnet/maui/main. Maybe @mgungorchamp can comment if .NET8 fares any better?

2. I did see @mgungorchamp's logs, where the Mono runtime appeared to crash on arm (32-bit) devices in .NET 7. It feels like it could be a runtime issue.

3. I'm not able to observe the app crashing as @mgungorchamp is. However, the best I have are older Pixel devices that are arm64.

To move further on this, there are two paths @mgungorchamp can take to help us investigate further:

1. Try .NET 8.
2. Try to create a smaller sample with the same issue -- @mgungorchamp basically sent their entire application where you have to navigate a web application to create an account, upload photos, to even use the app.

If we had a smaller repro, it would help me investigate any memory leaks (as there should be objects less in memory) -- and we would have something the runtime team would have an easier time investigating. Thanks!

[ghost]

Hi @mgungorchamp. We have added the "s/needs-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[akhanalcs]

@mgungorchamp Can you try the 2 paths that Jonathan recommended? I'm quite interested in this issue.

[akhanalcs]

@mgungorchamp Any update on this?

[mgungorchamp]

I paused the development... thank you for the follow-up, though!
Update
1- .NET 8. did not solve my problem - it crashed after several hours
2- Smaller app size - I need to find time to implement and test -

[Zhanglirong-Winnie]

Verified this issue with Visual Studio Enterprise 17.10.0 Preview 1. Can repro on Windows platform with sample project. Memory leak causes VS to crash.
https://github.com/mgungorchamp/InfoBoard/tree/Semaphore-NoSingletonNoRecursive/InfoBoard