How can Sentry improve Issue states

[Dhrumil-Sentry]

06/20/2023 Update See latest update of where we are landing in this comment: #43039 (comment)

Initial Discussion

We at Sentry have been discussing “What makes an issue relevant to our users and when?”

As we talk to customers it’s become apparent that our users often have to resort to time-based searches and filtering to get to the list of issues they care about: New issues and Regressions

Ideally, it should be very easy to find new issues and regressions, but we’ve found that old issues continue to clutter Issue streams. These issues are errors that users expect to see intermittently but don’t ignore because there are worried about missing any potential spikes.

We think the solution is to help automate your triage by building the following:

• Improved issue states and automatic state transitions
• Building the ability within Sentry to detect spikes in the event count of an issue

Current Issue Stream:

Issues Stream after proposed improvements:

How would such a solution work?

In addition to Resolved and Ignored, a Sentry issue will have the following additional states and transitions :

1. New
   1. An issue is New if it’s first seen in the last 7 days
2. Ongoing
   1. A New issue will change to an Ongoing issue 7 days after it’s first seen.
   2. If you have not ignored or resolved an issue after 7 days we think it's likely that you want to keep an eye on the issue but don’t consider it to be an urgent problem
3. Archived(Auto-Ignored)
   1. An Ongoing issue will become an *Archived(*Auto-Ignored) issue 30 days after it’s first seen
   2. If you have an issue that you did not resolve or ignore in 30 days- you either have too many issues or you’re avoiding Ignoring the issue because you are worried about missing escalations in event counts
   3. Sentry will keep a track of such issues and bring them to your attention if they escalate in event counts
4. Escalating (Any feedback on the naming is much appreciated)
   1. Sentry will automatically monitor Archived and Ignored issues to detect any spikes in event counts and if a spike is detected then the issue will become an Escalating issue
   2. You don’t need to worry about missing any potential spikes since Sentry is on watch for that so you can ignore intermittent issues at ease
5. Regressed
   1. If you resolve an issue and Sentry sees a new error for the resolved issue then it will become a Regressed issue
   2. A Regressed issue will become an Ongoing issue 7 days after Sentry detects the regression

Sentry will support searching New,Escalating and Regressed issues using filters such as is:escalating

We could also allow users to configure the transition time between states if they find this automated triage flow useful.

Your feedback is much appreciated

We’d love to hear your feedback on any of the following questions:

1. When do you consider an issue to be spiking/escalating?
2. What issue states do you want to see and what automated transitions would you prefer for your workflow?
3. Any feedback on the naming of these proposed issue states
4. Any other thoughts on how the Issue List area in Sentry can become more helpful for you, and why?

[mitsuhiko]

I really like it :) Some other thoughts I had on this in no particular order:

• 7/30 days is rather long but I can see where those numbers are coming from. If we stay true to our review flow, then maybe it would be interesting to move an issue to "ongoing" through explicit review. I don't quite want to ignore an issue straight away, but telling Sentry that I have acknowledged it in some form so that it keeps an eye out on escalating would be nice.
• Related to the earlier point: escalating sounds very useful, but in this proposal only after 7 days does an issue become eligible to escalate.
• I think the term "archived" has quite a bit of meaning and in some ways this appears to be what we consider ignored today. Maybe just literally calling it a form of ignored (eg: "auto-ignored" or "ignored due to inactivity") might be better?
• I always find it odd when a search mixes things together. That we refer to "regressed", "escalating" and "new" as is:escalating in the search feels odd. It sounds like we're somewhat dividing the issue stream anyways into two rather large pieces here: what we want to show, what we feel like is not valuable. Maybe this could be seen like issue open/closed? is:open, is:closed or is:active, is:inactive?

[robinrendle]

1. The search terms will apply to the states:is:new, is:escalating, is:ongoing etc.
2. Archived is auto-ignore, yes, but if I’m looking at a bunch of issues I want to quickly see which issues are "stale" vs. those I have explicitly told sentry to shut up about. Labelling them in this way will help with that.
3. Based on the issues I’ve looked at internally, Escalating is very rare and usually happens to old issues. We can improve how Escalating is determined in the future (maybe even adding manual controls!) but for now keeping the state changes simple is more important.

[Dhrumil-Sentry]

@mitsuhiko Thanks a lot for the feedback

1. Def agree that 7/30 days would not work for every project- this is something that could also be determined based on how many issues a project sees in a given time window. But the suggestion to allow users to move states from new to ongoing is helpful- Users apply their judgment that a new issue will keep happening and mark it ongoing directly

2. I def get your point new issues can escalate too

3. Wrt the searches @robinrendle has clarified that these would be different filters

[jmduke]

I think this is a net improvement over the current state of things, and am broadly positive about the proposal!

To open with an explanation of what my Sentry workflow is (aspirationally, so like 90% of the time) like:

• I try to keep For Review as inbox zero as possible.
• If something is a known one-off (ie its due to something I did or something I have already fixed) I resolve it.
• If something is a known issue that I do not consider urgent (a peaceful bug that I consider non-urgent, for instance) I link it to a GitHub issue.
• If something is known peaceful with no plans to resolve it, I ignore it.
• Every week or so, I go through "All unresolved" to resolve any issues that I believe I have fixed or are outdated, specifically so I can be notified through "For Review" if they recur,

I recognize that this is probably a non-universal and idiosyncratic use of Sentry, so with that proviso out of the way:

• I'd love to understand better the interplay between the expanded state space of an issue and its state of 'reviewal'/'nonreviewal'. My workflow with Sentry revolves around keeping inbox zero for For review and not for All unresolved — which transitions in this state machine, if any, would bring issues back up for review?
• Along those lines, one piece of issue state that isn't reflected in this proposal is linkage. Linkage is my way of reifying "okay, this is a problem, but it's not urgent" within Sentry. In my ideal world, that reification would trump any sort of opinion Sentry imposes upon the state of the issue.
• I agree with @mitsuhiko on 7/30 being somewhat surprising defaults, though I'm sure this varies pretty wildly from org to org. I'd like for these to either be configurable or manually transitionable.

I'm honing in on the state transition piece a lot because I have quibbles, but I want to close with how compelling this sentence is to me:

These issues are errors that users expect to see intermittently but don’t ignore because there are worried about missing any potential spikes.

This is a real thing, and really useful — especially at $PREVIOUS_COMPANY where Sentry hygiene was extremely low. Separate from all the "is this archived or ongoing or not?" bikeshedding, a simple view into "which issues, if any, are starting to crop up at a much higher cadence than normal?" is very high-value IMO.

[Dhrumil-Sentry]

Hi @jmduke Thanks a lot for this detailed feedback!

1. One of the things we've considered for triage improvements is to show you separate lists for New, Escalating, Regressed and Ongoing issues - We have some thoughts on how Sentry should show you a more compreshensive "Inbox" view. but would love to hear what Issue states would you like to see in your For Review tab

2. We agree that 7/30 days may not work for every project - What would be a preferred default for your workflow? The idea to let users manually transition the states after they review is really valid

3. Could you please expand a bit more on what you mean by Linkage ? Would automatic states and transitions like Ongoing, Archived and Escalating help achieve this? You see a new issue that you think is not urgent and you manually transition it to Ongoing (or Sentry does it for you), and Sentry will tell you if the issue sees a spike

Or is Linkage like a reminder to yourself that you need to see this issue again after a while?

[jmduke]

1. I think just escalating, regressed, and new. My understanding is that means it'd give me a view of everything whose potential urgency has changed since my last review cycle.
2. For me personally — probably 24 hours / 7 days respectively.
3. Yeah, sorry — linkage I just mean as "an issue has a concomitant GitHub Issue that was generated and linked to by Sentry". For my workflow, that's the representation of "this is an issue that I've reviewed, created a backlog item for, and plan on resolving".

[nanorepublica]

I generally like this idea especially as it would probably make configuring alerts easier as I generally never get beyond alerting for new or regressed issues, the escalating status would be helpful here.

I think my only comment is for issues going from 'ongoing' to 'archived'. Being able to keep an issue in ongoing would be useful.

An example of this is where I use Sentry for logging specific error events and one of my clients was doing a migration that covers a couple of months so tracking these events that were expected over this time was useful and would have been confusing to have disappeared into archived.

I would also expect to be able to resolve an issue from any state as I also typically tend to resolve all issues to have a clean slate to see what is actually still a problem with whichever codebase I am working on.

[Dhrumil-Sentry]

HI @nanorepublica Thanks a lot for the feedback!

We definitely agree that developers would like to have some configurability to make these transitions work for their preferred workflow and that is something our team would iterate on as well.

You will still be able to resolve or ignore New/Ongoing/Escalating/ Archiving issues, We think of these new states as additional improvements to the current Unresolved state

[dcramer]

fwiw i would not expect ignore and archive to exist - they're the asme status with different names. as a user this would be extremely confusing to me if i have two things that literally achieve the same result

for keeping things in ongoing, maybe archived is configurable? its just auto-ignore after all. we've had auto-resolve in the past and it was configurable. there's probalby something we can learn from the lessons of its implementation and usage.

[eric]

Is there work being done to think about the ways deployed software operates differently than a web service?

If I’m still currently getting crashes for an issue and it’s only happening for an iOS release that happened a year ago, that’s a useful signal.

[dcramer]

would you want to auto-ignore it? deprioritze it? escalate it in that case (assuming its new)?

[eric]

We have many crashes that may happen once or twice a day or maybe even once or twice a week. Being able to understand if there was a release where we fixed the issue is not currently easy to do.

Generally being able to quantify "this is a new issue" vs "this is an ongoing issue" vs "this is an old and fixed issue" would have helped me to not spend time trying to fix an already-fixed issue.

[zoesyc]

Hey @eric - we are absolutely investigating ways and processes where deployed software (e.g. mobile) differs from web!

With our existing workflow, you can tie 'ignored issues' to a number of occurrences (eg. 100 times per week) and 'resolved issues' to a given release (e.g. 1.2.3) such that if an issue occurs that goes beyond the specified bounds (e.g. 200x in a given week or release 1.3.4) than it would be a truly new issue.

What I'm curious to understand is do you make use of the current triage workflow or even if it's clear? Subsequently, are there other key signals you find useful that should be highlighted more prominently?

[nanoburd]

Thinking about performance issues that I may resolve.

Some nuances to consider:

1. When I push a fix for a performance problem, I may not see immediate effect. I may only see it improve several days later.

• this should be covered with your ongoing and archived status
• we need triggers to monitor these issues to see if these issues improve over time but not resolved

2. I may also partially improve the performance but its not enough to solve the entire performance problem. There are occasions that many different strategies need to be employed to improve performance.

3. Performance Problems are a function of time not number of events

• So maybe part of Archived, but performance issues need to be monitored for both improvements and worsening over time, as the impact or severity of it can change as a function of time.
• Where an error this is captured by escalating. A performance issue might still have a low number of events, but the actual problem itself is getting worse.

[dcramer]

When I push a fix for a performance problem, I may not see immediate effect. I may only see it improve several days later.

I think our current resolved in release would still solve for this in the situations Im thinking of

[nanoburd]

If we consider perf issues as binary (solves or a problem), then yes resolved+regressed works great. We've artificially created this with the thresholds with each performance issue detector.

In reality when developers think about performance their target could be some X% improvement, and they want to track (maybe report) that result. Despite it no longer triggering our threshold. This could be dependant if its an important endpoint etc for them. This also helps them promote their impact on dedicating time to performance which often times is deprioritized.

Whether its a state or some sort or feature to track success, I think it matched the mental model as developers engage with the performance problems on their application.

[Dhrumil-Sentry]

I think @nanoburd has rightly called out that these issue states and transitions may work well for errors but they don't improve the workflow for other types of issues.

But should these gaps be addressed by Issue state improvements- I am not sure. I could imagine alerts wherein Sentry can inform developers when performance issues worsen (Although I know we'd like to avoid more notifications)

But our issue workflow is definitely not suitable for tracking gradual improvements- A possible solution could be to tie in improvements made to fix a performance issue to the Performance view itself

[dcramer]

I think we're talking about issue types that dont yet exist. All of our current issue detections are stateless - they dont require delta between before and now - but even in stateful issues, you can tell the system when its resolved. if its resolved when X infrastructuire in place, yeah we're not doing that, but if its something literally in the application code, which is our focus, then i still am not finding a case where the code change reference is invalid.

i would agree that if all we can do is say "the website is slow" none of our features work, but the entire product is useless at that point when it comes to Issues

[seanjfellows]

This should be tremendously useful for us.

We're currently working very hard to keep Sentry at Inbox Zero, primarily by leveraging all of the smart ignore options. In essence we are simulating the "escalating" feature by ignoring many Issues until they reach a new threshold.

Put another way, we consult Sentry using two different mindsets:

1. Short term planning, looking for urgent items. In the language of this ticket, we are looking for new and escalating issues.
2. Longer term planning, looking for tech debt and old items. In the language of this ticket, we are looking for any issue, including/especially ongoing and archived.

Today we have to keep Inbox Zero to accommodate (1) and it is very tricky and nobody has much confidence that we are being consistent.

If Sentry were opinionated about this, I think it would help a lot.

[Dhrumil-Sentry]

@seanjfellows Thanks a lot for the feedback, Would be really helpful to know when do you guys think an issue is "escalating"? Is it some x multiple of the event count seen daily? Any details on how you folks use the current ignore options would be valuable to know.

[seanjfellows]

If I said "I know it when I see it", that would be both unhelpful and honest. :)

But to try a little harder, I think at least some of these are factors:

• Sudden increase in event count, relative to the past for this issue. e.g. it went 10x in an hour or a day.
• Increase in event count over any time scale, relative to total volume for the project. e.g. it has slowly grown to be one of our top Issues.
• Increase in diversity of one of its tags. e.g. it used to just have a tag with value X but now it has expanded into X+Y. This would be especially useful for some of our custom tags or the user tag.

Regardless, IMHO, the thing that is more important than the actual logic is clarity on what the logic is. Put another way, it is much easier to triage an issue that says, e.g., "Sentry says this is escalating because it is now affecting 1000 more users than before" than an issue that says "Sentry says this is escalating because of magic". If I have the rationale in hand, it is easier for me to judge "Ok, does this actually make sense? Do I actually think this is now urgent?"

[therealarkin]

@seanjfellows that's a good feedabck @trillville @brianthi for "Escalation Reason".

[robinrendle]

Hi everyone, here’s an update.

In this first round of changes we’re planning on replacing the “Ignore” action with ”Archive”: from here on out you’ll be able to quickly archive issues if they’re not important. But now Sentry will automatically bring the issue back if it escalates and a high number of events happen in a short period of time. This is particularly handy if you want to Archive all the issues in a project to get to inbox zero and let Sentry tell you when anything gets worse.

Hit the Archive button in the top left, and you’re good to go:

The Archive action will also replace Ignore on the Issue details page, but if you want to archive an issue forever (just like how Ignore works today) you can find that option in the dropdown:

Escalating issues will move out of the Archive tab automatically and so you’ll find them in the Unresolved tab along with New and Regressed issues. Notice the issue at the very top here:

Click into an escalating issue and you’ll see an explanation of why we thought it was escalating, with the chance to offer some feedback:

And if you want to find all the Escalating issues across your projects then they’re just a click away in the tabs on the Issues page:

Don’t worry about all those issues you’ve archived as they aren’t hidden away forever, you can always find them in the tabs, too. In future updates we’re planning on automatically archiving issues older than 30 days so that you can focus on the issues that matter most to you: New, Regressed, and Escalating issues.

And that’s about it! We’re excited about these changes because we want to quickly show you relevant issues without you having to hunt for them. And these changes are just the first step.

Let us know if you have any suggestions or feedback!

[seanjfellows]

What's the difference between Resolved and Archived?

Am I reading this right that the difference is:

• Resolved: I assert this is fixed [as of release X]. Let me know if that is not true, in which case I want to know ASAP because it is important to know I was mistaken.
• Archived: I assert this isn't super important right now. You might keep seeing this or you might not. Let me know if it gets worse, in which case I want to know next time I look at my review items.

[armenzg]

What @seanjfellows says.

In my own words: Archived means that it is still an ongoing issue while Resolved means that we expect it to be over.

[Triplkrypl]

Nice idea for some automated threshold for ignored issues, because figure out some suitable value take time, we have issues still going back from ignore list. Mainly issues caused by client system integrations of our API.
I only do not understand why is ignore function renamed into archive, we really not care if ignored issues happens if they are in normal traffic rate (otherwise increase of ignored issues can means for us some other failure or attack attempt and is good to know).
Name "archive" can be misunderstand as issue should be preserved for ever and not auto deleted after 90 days.

[therealarkin]

@Triplkrypl can you elaborate why the name "archive" is confusing?

Re: 90 days - this is the case w/ all issues without events. If events keep happening, the issue will stay.

[chadwhitacre]

can you elaborate why the name "archive" is confusing?

I think he said it: "archive" could be interpreted to mean that the issue will be preserved for ever, i.e., not auto-deleted after 90 days—same sense as in the GitHub Archive Program or the Internet Archive, the point is permanence.

[therealarkin]

Following this discussion and after extensive testing with hunderds of customers, we're ready to share our updated plans for issue states. Your feedback has been vital in shaping these changes, and we hope they will improve the experience for all of our users.

Here are the main points of our proposed updates:

1. "Mark reviewed" button: Although not widely used, we're retaining this feature to maintain existing workflows and manually change transitions. Several organizations were actively trying to opt out of the beta due to its removal. We've heard your requests via email and on this GitHub post to manual state transition. When a user marks a “New issue” as “marked reviewed”, it will transition into “Ongoing”.

2. Extension of the "new" status to 7 days In our beta, this was set at 3 days. However, we've decided to extend it to better reflect the typical lifecycle of an issue, taking into account weekday seasonality and the need for at least 7 days of historical data to generate accurate forecasts for escalation (see next bullet).

3. Eligibility for every ongoing issue to be marked as an escalation issue: We're moving away from relying solely on Archive-until-escalating to run our escalation algorithm. The concept of escalation is relevant to any ongoing issue, and we're making this change to reflect that.

4. Halt on the release of auto-archive: Our priority is the sorting feature (which will share some thoughts in a separate discussion post), and after considering internal feedback, we believe it's less crucial to automatically remove older issues. We're pivoting to focus on streamlining auto-resolve, as suggested in this discussion. We will share details later here.

5. Issue stream tabs changes: We've refined the tabs to include "All Unresolved", "For Review", "Regressed", "Escalating", and "Archived". We feel the addition of "Regressed" will add value, and since we have “All Unresolved” & “For Review”, we believe the "Ongoing" & "New" tabs are not necessary.

6. Removal of the escalation floor in the algorithm and more transparency: This minor but vital change addresses an issue where currently, an issue with less than 30 events per hour only escalates if it has 200 events per hour. This doesn't deliver on our promise of Sentry watching your back, and we're changing that. You can read more about the algorithm here

Here's an updated diagram of the issue states. Please let us know if you have feedback or questions. Thank you for being part of this journey!

[rachrwang]

Big thanks to everyone who shared their feedback with the team - looking forward to these updates!

[armenzg]

Thanks for keeping us in the loop! Looking forward to it!

[przemekpastuszka]

Hey 👋

Is it possible to turn off autoescalation for issues archived until...?

What's my problem:

• I've got an issue which I risk accept, as long as it happens <100 times a day
• I mark the issue as Archived until it occurs again for 100 times
• the issue goes away for some time (no error events for let's say a week)
• but then the issue reappears, let's say 80 times in a day, and the issue is marked as Escalating.
• that's not the outcome I was looking for, as I don't care about variability of occurrences, only that it's less than 100 times a day

This used to work just fine before introducing escalation algorithm, but it's broken now. How do I fix that?

[Dhrumil-Sentry]

Hi @przemekpastuszka Thanks for this feedback ! @brianthi @rachrwang Is this this expected behavior ? I assumed that if a user marked an Issue as Archive until x then the escalation threshold would be what the user has setup?

[brianthi]

Hi @przemekpastuszka, I think you may either be setting the archive until option incorrectly, or you've found a bug. Could you create a support ticket here, and give the team a link to the issue that you saw this behavior? We can continue to help you there.