Mobile Device Classification

[edwardgou-sentry]

Hi! I’m an engineer on the Mobile Developer Experience team at Sentry. We've heard from some of our mobile developer users that they find it difficult to understand the impact of issues and performance of their mobile apps across the thousands of unique mobile device models available today. Recently we added a mapping for 18k+ device hardware identifiers into actual device product names to help you better understand which devices are being affected by errors and performance, but we know there’s more to be done here. We're looking for feedback from the community on how we can make Sentry more effective in this area. Here are some questions you can answer to help us understand the problem better:

• How do you ensure your application best supports the thousands of devices available in the mobile market today? What are the challenges you face?
• What device specs or criteria do you most care about when debugging issues in your mobile app? For example, cpu frequency, core count, frame rate, screen size, etc.
• At what stage of the issue triaging process, if any, are you considering the range of devices that are experiencing the issue in order to determine if it needs further attention and how does it affect your prioritization? What do you consider before, or after that?

One concept we’re considering is to classify devices into categories of high, medium, and low performance based on the client mobile device under a new searchable tag. Here are a couple of ways we’ve thought about implementing this:

• Android 12 and beyond introduced a Performance class api we could possibly leverage:
  • Performance class API
  • This doesn't address iOS devices, we'd have to use a different strategy for iOS
• Or classify using heuristics based on device memory, core frequency, number of cores, etc.
  • Some form of this already exists in Sentry

Some questions we have about this approach are,

• Would this approach improve your ability to triage issues with Sentry? If not, why?
• Are there specific hardware criteria you value more than others when classifying devices
• Do you have any other concerns or feedback about this approach?

Please leave us any thoughts or feedback on what you’ve seen here! Thank you!

Have any other feedback you’d like to share? You can always find us on Discord or submit an issue on GitHub.

[phacops]

This classification is very helpful with Profiling. We present a lot of aggregated data to our users and aggregating various types of devices is not always useful because it averages everything too much. It's been helpful to track down issues within those groups. For example, highlighting that a write on disk for a low end phone was taking a long time compared to the latest iPhone, an issue we might have not been able to surface if we took the p95 of the function for every execution of the functions.

As we work on detecting profiling issues, we'll likely keep finding that useful to push issues that could only be for one group of devices.

[markushi]

I'm wondering if we could use one of our own metrics. E.g. our mobile SDKs automatically generate a app.start.cold span. If we group those spans durations per device we could get a ranking. There are some implications around this (it's customer data, potentially incomplete, ...) but probably still worth discussing.

[edwardgou-sentry]

Hey @markushi, interesting idea! I thought about this a bit, a couple of tradeoffs I think we might encounter here:

• We probably wouldn't be able to capture all devices in the market today since our sample size is limited to what data is available in Sentry. Newer devices would also need to be periodically added, so some level of maintenance would be necessary for this approach.
• I'm not sure if we'd be able to come up with an unbiased ranking. Metrics such as app.start.cold are also affected by quality/nature of the app itself, so this might skew results in the ranking. I could be wrong here though, I'm not sure how big of a factor actual app performance plays into this vs device performance for the metrics we capture.

With that said, do you feel that app.start.cold (or are there any other metrics captured by sdk) is representative of a device's performance level? We're thinking about what criteria to consider when determining devices classes, like screen size, core frequency and count, memory, etc. It would be great to know what metrics or specs are most important for you!

[markushi]

Thanks, that's some good points! I think in general app.start.cold would be a good/representative metric as it captures a typical workload and requires a mix of good disciplines (disk + memory + cpu). But you're right: the metric is bound to a specific org/app/release.
I think CPU cores/freq is simple and translates well to device speed. But for screen size, memory and disk it's trickier, as having more doesn't mean the device itself is fast.

[edwardgou-sentry]

Hey everyone, just wanted to give an update on what we're moving forward with. Here's the criteria we decided on:

apple:
    < 2000mhz => low (class 1)
    < 3000mhz => med (class 2)
    otherwise high (class 3)
android:
    < 8 cores or < 2000mhz or < 4gb memory => low (class 1)
    < 2500 mhz or < 6gb memory => med (class 2)
    otherwise high (class 3)

This is mostly the same criteria as the existing device classification for profiles in Sentry at the moment, with a few small adjustments for android devices. We checked this criteria against a number of old and new devices of different performance levels available today and we're satisfied with the output.

Also, one important implementation detail we've settled on is to store device classes as numbers (ie 1, 2, 3) instead of explicitly high, medium, low values. This allows us to scale higher grade device classes as mobile devices naturally increase in performance (ie 4, 5, 6 and onwards) and we can shift our mapping of high, medium, and low as a sliding window when necessary.

[romtsn]

@edwardgou-sentry a little bit late to the party, but here's how Telegram does classification - perhaps, we could derive something from there. E.g. they also consider Android API version, and there's device's memory class available.

[simone-lingokids]

Hi @edwardgou-sentry!
I have a couple of questions about this classification:

• How do you calculate the frequency when you have multi-cores with different frequencies?
• From the data we got in the Sentry instance we use, I see that the device.class for Android models is empty, while it's almost always present for iOS models. Is it an issue I should report, or is there something we need to do in our config?

[shaedrich]

screen size […] the range of devices that are experiencing the issue in order to determine if it needs further attention

I think, especially for web apps, this is quite important. However, in combination with different browsers and versions, tags might not be the ideal place for more in-depth information, since this can quickly become bloated. What about a new section below the tags with a table listing the most important "features" and headers (e.g. Client Hints) (similar to what you already do in the "Operating System" section down below). The importance might be determined/weighed by

• Settings
• "Classification tags"
  • desktop/web/mobile
  • frontend/backend
  • etc.
• AI for issue-specific points of interest

AI might even be able to point out if the manufacturer (browser, os, device) is currently experiencing any issues that might cause this very issue.