SIGTRAP Android freeze

[LeviWilliams]

Description

Recently we upgraded our Skia version from 0.1.197 to 0.1.214 and now we are seeing a bunch of "SIGTRAP: Trace/breakpoint trap" in production on Android devices with a variety of versions. Currently on react-native 0.72.3 if helpful.
.

We reproduced a couple times on a Pixel 6 and the app just freezes. I understand this error is vague, we currently have no leads as to why this happens after updating the lib though if we are able to provide a repro we will.

Let us know if there are any ideas on what we can try, thanks for the help as always.

Version

0.1.214

Steps to reproduce

Snack, code example, screenshot, or link to a repository

[wcandillon]

We would definitely need a reproducible example and also a sense of the APIs which are used (Reanimated version, Skia animations, etc)

[laurens-lamberts]

Hi @wcandillon, @LeviWilliams,

We also experience this crash in production. So much - unfortunately - that we received a warning from Google regarding 'Android vitals bad behavior'. Our app will become less discoverable and receive a warning at the store page if this crash is not resolved soon.

We currently have no clue when/where specifically this crash occurs. Therefore we cannot provide a reproducible example at the moment.
For users that experience the crash, we do notice that this crash is experienced only once every app-update. Next sessions are not affected most of the times.

For our app, in the last 7 days 1.5k users experienced 1.7k crashes of the SIGTRAP type, originated from librnskia.so.

Two crashes occur, both indicated by SIGTRAP, both about 50% of the total occurrences;

[split_config.arm64_v8a.apk!librnskia.so] SkTDPQueue<GrGpuResource*, &GrResourceCache::CompareTimestamp(GrGpuResource* const&, GrGpuResource* const&), &GrResourceCache::AccessResourceIndex(GrGpuResource* const&)>::remove(GrGpuResource*)

and

[split_config.arm64_v8a.apk!librnskia.so] GrResourceCache::notifyARefCntReachedZero(GrGpuResource*, GrIORef<GrGpuResource>::LastRemovedRef)
SIGTRAP

Some more details

100% foreground crashes, spreaded to usage over Android versions and devices.

Versions used in the context of the above;
Skia: 0.1.221
Reanimated: 3.5.4
React-native: 0.72.7

With Skia 0.1.210: No significant crashes
With Skia 0.1.214: many SIGSEGV crashes, and also some SIGTRAP crashes
With Skia 0.1.221: many SIGTRAP crashes, no more SIGSEGV crashes.

Hope this helps tracking down the issue.

[wcandillon]

@laurens-lamberts Thank you for this precious data.And I hope that we can get this sorted out as soon as possible. I need to review things more carefully on my side but from 0.1.210 to 0.1.214, the only update that I am seeing on the native code is the Skia version upgrade. I could do another upgrade to see if this helps. (alternatively we could downgrade as well to see if this helps).

The error seems to be Skia specific, I will investigate this a bit deeper and let you know if I find anything.

[wcandillon]

Strangely enough, I cannot find any relevant change from 0.1.214 to 0.1.220.

Going forward, we will setup some of release program to check if we introduce such regressions to releases.

We have a RN Skia client of approximately the same scale as you running 0.1.213 (has no crash reports). I will contact them about the issue and see if there is a way to maybe just try to upgrade to m119 in an isolated manner.
This client is not using any of the recently deprecated APIs.
The holidays may make things a bit slower there but I will report back.

@laurens-lamberts I suggest we do the following:

• I'll keep doing surface level investigation to see if we can get any interesting lead
• Let's upgrade the app to use Skia m121 and also not use the deprecated APIs and see if the crash still occur.
• If it does, we will make a custom package that is the latest RN Skia but uses m116 (I believe it wouldn't be an excessive amount of work to make such package).
• If that fixes, the issue, we will discuss on how to proceed.

[laurens-lamberts]

Thanks a lot @wcandillon, we're really happy with your support proposal on this issue.
It means a lot to us, and we are motivated to help tracking down the issue.

Due to the high impact when issues arise during deployment in the christmas / newyear period, we will postpone next releases to January. For the upcoming release of our project we upgraded to the following library versions (all latest);

"react-native": "0.73.1",
"react-native-reanimated": "3.6.1",
"@shopify/react-native-skia": "0.1.230",

If any new versions of the above packages appear before our release, we will update to ensure having the latest of all.

We always perform our releases phased, so as soon as we got insight in crash rates we will share them with you. This will likely be the end of January / beginning of Februari.

For my information, where in the react-native-skia library do I find the reference to the internal Skia version number (like m121)?

[wcandillon]

This is were you can find the Skia version used : https://github.com/Shopify/react-native-skia/blob/main/.gitmodules
In the built package, I don't believe this information is available, that something we could do potentially if you would find it useful.

I will continue to investigate this a bit and also do the upgrade to m121 and we can tackle this more aggressively after the holidays. I think that we are lucky to have a Skia client that uses 113 at scale but with only non-deprecated API, that will give us a lot of information once/when they deploy 114 and above.

[laurens-lamberts]

Yes, that's great. Looking forward to hearing their experiences with later versions.
Thanks for showing me where to find the Skia version used. Maybe we can use that in combination with the release notes of skia to troubleshoot some issues in the future.

[espenjanson]

Any updates on this/ways to resolve it @laurens-lamberts @wcandillon @LeviWilliams ? Anything we can do to help? We just had to downgrade Skia to 196 because of thousands of crashes in production due to this error. Would be awesome to be able to upgrade since we want to move on to RN 0.73 (which according to release notes are not fully supported until 213) 🙏

[wcandillon]

@espenjanson Yes anything that would help to reproduce the issue or more details on the conditions of the crash would be extremely useful. I'm surprised you are on 196 because 197 notoriously fixes a crash related to animations.

We have been coordinating with @laurens-lamberts to find the root cause of the issue but without success yet. We have a large client who's running the latest version of Skia without any crashes (this same client had a large amount of crashes in production with 196). This means that the issue is likely related to a particular API but we haven't been able to identify it yet.

[wcandillon]

@espenjanson could you send me a list of Skia APIs and components you are using? You can do it privately as well by email.

[espenjanson]

@wcandillon thanks for quick response. We'd love to help in any way we can. Any chance you could provide the package.json (or at least parts of it, such as react-native and reanimated version and perhaps other libraries that could affect skia)?

If you want to, we can send you a minimal functioning app project with our crashing package.json and all the components we have that use Skia. Can put together a zip or a repo, whatever works better for you. If needed, we can also provide more detailed stack traces from Google Play and Sentry.

Will put the team on this immediately. Thanks a million for paying attention to this!