Crash Reporting with Sentry
Firefox for iOS uses Sentry Cloud for crash and exception reporting. This kind of reporting gives the Firefox for iOS team valuable insight as to why the application crashes or behaves incorrectly. It is one of the key methods we use to improve the product in terms of stability.
This page explains how Sentry works, how the various parts interact and what kind of data it sends back to the Firefox for iOS team.
Sentry is an open source crash reporting and aggregation platform. Both the client SDK, github.com/getsentry/sentry-cocoa, and the server, github.com/getsentry/sentry, are open source.
On the client side Sentry is invisible. There are no parts to interact with. It reports crashes and fatal errors back to Mozilla in the background. Sentry is enabled when the Send Anonymous Usage Data switch in the Firefox for iOS settings is enabled by the user. By default this switch is enabled, it is an opt-out mechanism.
On the Sentry Cloud website there is a dashboard that the Firefox for iOS team uses to look at incoming crash reports. The dashboard lets us inspect the crash report in detail and for example see where in the application the crash happened, what version of the application was used and what version of iOS was active. Below is an overview of all the attributes that are part of a crash report.
A typical Sentry crash report contains three categories of data: device, application, crash.
Sentry collects basic information about the device the application is running on. Both static (device type) and dynamic (memory in use, boot time).
model_id: "J82AP",
family: "iOS",
arch: "arm64",
storage_size: 31989477376,
free_memory: 206585856,
memory_size: 2084569088,
boot_time: "2017-07-22T20:24:34Z",
model: "iPad5,4",
usable_memory: 1860943872,
type: "device"
Sentry collects basic information about Firefox for iOS. The device_app_hash is a Sentry generated identifier that allows it to group crash reports for a specific client. This identifier is unique to Sentry and is useless outside of Sentry. It cannot be used to correlate a specific user to a crash. It is also not related to any identifiers that Firefox for iOS uses internally.
executable_path: "/var/containers/Bundle/Application/07A66DB3-33D8-4C98-ABE6-AE3571A0189C/Client.app/Client",
app_identifier: "org.mozilla.ios.Firefox",
device_app_hash: "971aee19c94f2ec9b35518973bd306020745cabb",
build_type: "app store",
app_start_time: "2017-07-25T13:57:23Z",
app_version: "8.0",
type: "app",
app_build: "4558"
Every crash report contains a reason - why did this crash happen. This field can contain two different values:
- an error message generated by iOS
- an error message generated by Firefox
Both Apple and Mozilla make sure that no personally identifiable information is put in any of these messages. We keep them technical and to the point.
Example of an iOS generated message:
NSInternalInconsistencyException - fatalApplication threw exception NSInternalInconsistencyException:
attempt to delete item 1 from section 0 which only contains 1 items before the update
Example of a Firefox generated message:
BEGIN EXCLUSIVE failed. Error code: 0, Error Domain=mozilla Code=0 "Non-open connection;
can't execute change." UserInfo={NSLocalizedDescription=Non-open connection; can't execute change.}
Every crash report contains a list of libraries and frameworks that the application links against. These include: Frameworks written by the Firefox for iOS team, Third-Party frameworks that we link against and finally system frameworks.
The UUID fields in these image descriptions are the same for every copy of Firefox for iOS and are not connected to a specific installation, device or user.
cpu_subtype: 0,
name: "/var/containers/Bundle/Application/07A66DB3-33D8-4C98-ABE6-AE3571A0189C/Client.app/Client",
revision_version: 0,
major_version: 0,
image_vmaddr: "0x100000000",
image_addr: "0x100060000",
minor_version: 0,
cpu_type: 16777228,
image_size: 3571712,
type: "apple",
uuid: "D16EC59F-2361-3621-A62F-AB41F5F0F869"
cpu_subtype: 0,
name: "/System/Library/Frameworks/Accelerate.framework/Frameworks/vImage.framework/vImage",
revision_version: 0,
major_version: 331,
image_vmaddr: "0x1826db000",
image_addr: "0x18f347000",
minor_version: 5,
cpu_type: 16777228,
image_size: 2732032,
type: "apple",
uuid: "1F670947-59DE-3818-9883-0A3692EDFF0F"
Every crash report contains a stack trace, which shows what functions in the Firefox for iOS code led to this crash. It includes names of iOS system functions and Firefox for iOS functions.
0 CoreFoundation 0x30f7ccfe0 __exceptionPreprocess
1 libobjc.A.dylib 0x30cdb4538 objc_exception_throw
2 CoreFoundation 0x30f7cceb4 +[NSException raise:format:arguments:]
3 Foundation 0x310d7a720 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:]
4 UIKit 0x31c446d24 -[UICollectionView _endItemAnimationsWithInvalidationContext:tentativelyForReordering:animator:]
5 UIKit 0x31bd09f1c -[UICollectionView _updateRowsAtIndexPaths:updateAction:]
6 Client 0x2000c9e9c specialized TopTabsViewController.((reloadData in _3434E9307FC2B0225EF0EAC66FE81010)() -> ()).(closure #1) (TopTabsViewController.swift:454)
7 Client 0x2000c5304 [inlined] TopTabsViewController.((reloadData in _3434E9307FC2B0225EF0EAC66FE81010)() -> ()).(closure #1)
8 Client 0x2000c5304 partial apply for TopTabsViewController.((reloadData in _3434E9307FC2B0225EF0EAC66FE81010)() -> ()).(closure #1)
9 UIKit 0x31bb733dc +[UIView(UIViewAnimationWithBlocks) _setupAnimationWithDuration:delay:view:options:factory:animations:start:animationStateGenerator:completion:]
10 UIKit 0x31bcb7a24 +[UIView(UIViewAnimationWithBlocks) animateWithDuration:animations:completion:]
11 Client 0x2000c0150 TopTabsViewController.(reloadData in _3434E9307FC2B0225EF0EAC66FE81010)() -> () (TopTabsViewController.swift:463)
12 Client 0x2000c2e60 [inlined] TopTabsViewController.tabManager(TabManager, didSelectedTabChange : Tab?, previous : Tab?) -> ()
13 Client 0x2000c2e60 @objc TopTabsViewController.tabManager(TabManager, didSelectedTabChange : Tab?, previous : Tab?) -> ()
14 Client 0x2000c3084 [inlined] dynamic TopTabsViewController.tabManager(TabManager, didSelectedTabChange : Tab?, previous : Tab?) -> ()
15 Client 0x2000c3084 protocol witness for TabManagerDelegate.tabManager(TabManager, didSelectedTabChange : Tab?, previous : Tab?) -> () in conformance TopTabsViewController (TopTabsViewController.swift:497)
16 Client 0x20025945c [inlined] specialized TabManager.(selectTab(Tab?, previous : Tab?) -> ()).(closure #1) (TabManager.swift:192)
17 Client 0x20025945c [inlined] TabManager.(selectTab(Tab?, previous : Tab?) -> ()).(closure #1) (TabManager.swift:192)
18 Client 0x20025945c [inlined] thunk
19 Client 0x20025945c [inlined] specialized Sequence.forEach((A.Iterator.Element) throws -> ()) throws -> ()
20 Client 0x20025945c specialized TabManager.selectTab(Tab?, previous : Tab?) -> () (TabManager.swift:192)
21 Client 0x2000c1970 [inlined] TabManager.selectTab(Tab?, previous : Tab?) -> ()
22 Client 0x2000c1970 TopTabsViewController.togglePrivateModeTapped() -> () (TopTabsViewController.swift:215)
23 Client 0x2000c1a40 @objc TopTabsViewController.togglePrivateModeTapped() -> ()
24 UIKit 0x31bb73010 -[UIApplication sendAction:to:from:forEvent:]
25 UIKit 0x31bb72f90 -[UIControl sendAction:to:forEvent:]
26 UIKit 0x31bb5d504 -[UIControl _sendActionsForEvents:withEvent:]
27 UIKit 0x31bb72874 -[UIControl touchesEnded:withEvent:]
28 UIKit 0x31bb72390 -[UIWindow _sendTouchesForEvent:]
29 UIKit 0x31bb6d728 -[UIWindow sendEvent:]
30 UIKit 0x31bb3e33c -[UIApplication sendEvent:]
31 UIKit 0x31c338014 __dispatchPreprocessedEventFromEventQueue
32 UIKit 0x31c332770 __handleEventQueue
33 UIKit 0x31c332b9c __handleHIDEventFetcherDrain
34 CoreFoundation 0x30f77b42c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
35 CoreFoundation 0x30f77ad04 __CFRunLoopDoSources0
36 CoreFoundation 0x30f7789a8 __CFRunLoopRun
37 CoreFoundation 0x30f6a8da4 CFRunLoopRunSpecific
38 GraphicsServices 0x312b78074 GSEventRunModal
39 UIKit 0x31bba3058 UIApplicationMain
40 Client 0x2000695dc main (main.swift:16)
41 libdyld.dylib 0x30d6ca59c start
If you've read this far, you might be interested in learning more about some example of use cases where this tool was useful for the team. Here are some examples: