Workflow 2.0

[dcramer]

Our core workflow has begun to suffer over the years - from growth of technology, scale of data, and general lack of attention. Recognizing this, we had the following conversation this week internally, and are setting out to tackle this concern. That core workflow includes several components that are loosely coupled together:

• the release metadata which allows us to identify the version of code an event is present in
• the new alert (and other variations) approach to notifications
• the new deployment notification - which requires a good amount of effort to achieve correctness
• the resolved in release and ignore issue actions and the general triage flow
• the regression notification which is key to understanding if an issue remains unresolved

These all string together to create a workflow that is tightly coupled to how I - and I believe most developers - think about shipping code. We dry run a bunch of changes, pray our tests are accurate enough, ship the code, and inevitably find a problem. The faster Sentry can connect those dots with accurate diagnostics, the better the outcomes for our customers.

The challenge here is that there's a fundamental gap in the workflow in how we notify about issues. We rely on "New Issue" to be timely and contextual. That is, we duct taped a solution that assumed most of the time new issues happened with new code. That's not always true, and technologies like JavaScript have decreased the signal to noise ratio over the years. So let's tackle that problem. There's a few key things we should look at as part of this:

1. What does a timely notification look like connected to the release lifecycle? That should focus on how we help identify truly "new issues caused by code changes".
2. How do those notifications change (or become addititive) with things like code push or feature flags, where we're making behavioral changes but not shipping a new SHA?
3. How can we improve things like "Ignore Issue" to be more functional usable? We shouldn't rely on custom alerts, or custom saved searches for such common concerns.
4. "Resolved in Next Release" can be problematic in many environments, but we now have SemVer support. Can we leverage that better?
5. Where is fingerprint breaking down? Are there platforms that are not effective that we need to resolve our native heuristics?
6. How does this apply to performance? to csp? to other "problems" (ala "issues")? Issues needs to be a platform to enable the workflow.

Most importantly, with all this in mind, how do we resolve the workflow without requiring customers to configure anything? We've - for better, or more likely worse - taken the approach over the years to put this problem into our customers hands by giving them complicated solutions to create alerts, complicated solutions to improve fingerprinting, and we've stopped trying to build a first-class curated solution to the problem. This is our chance to correct that.

For the community, if you have opinions, what are they? What can we do better here? What works well? What is completely terrible? We already have some solid foundations, but if you've got an opinion, let's hear it!

[alexbjorlig]

In my perspective the key issue with Sentry is configuration and setup. I think the solution to this problem is for Sentry to carefully choose a set of technologies and platforms to support; and then really support thoose. A bit like how the adapter architecture with SvelteKit works. You develop a solution, but can essentially ship it anywhere as long as there is an adapter provided.

Thanks for being open about this - cool decision 😎

[chadwhitacre]

carefully choose a set of technologies and platforms to support; and then really support those

@alexbjorlig You mean, like, limit the number of SDKs we support? Sentry has a core value "for every developer" so that goes against the grain, but maybe there's more we can do to support community-supported SDKs? Or better clarify which are "officially" supported and which are community supported? What SDK/platform do you use Sentry with the most?

[alexbjorlig]

carefully choose a set of technologies and platforms to support; and then really support those

@alexbjorlig You mean, like, limit the number of SDKs we support? Sentry has a core value "for every developer" so that goes against the grain, but maybe there's more we can do to support community-supported SDKs? Or better clarify which are "officially" supported and which are community supported? What SDK/platform do you use Sentry with the most?

I think a fundamental change of perspective is needed. SDK's, APIS's and CLI's are nice, but not what really deliver value. It's the working configuration, that actually ships the correct artifacts to Sentry and makes source-maps from typescript work. And currently Sentry has no solution for this - the only answer is to carefully configure SDK's and CLI usage.

In my vision for Sentry, SDK's, API's and CLI's are tools that Sentry's employees (and expert users from the community) use to build adapters. An adapter would for example be:

Sentry for Next on Heroku
Sentry for Next on Vercel
Sentry for Next on Firbase
Sentry for Sveltekit on Heroku
Sentry for Sveltekit on Vercel
Sentry for Sveltekit on Firebase
Sentry for Vue on Heroku
Sentry for Vue on Vercel
Sentry for Vue on Firebase

I completely understand that there are almost infinite combinations. Howerver the benefit of such an adapter setup would be the configuration hassle is now on Sentry developers and not developers having to figure out a ton of things --> out of the box all essential stuff is configured.

I have personally dealt with the setup for source-maps for an Angular app deployed to Heroku, and a Sveltekit app deployed to Heroku. It's really not easy at all. An abnormal amount of effort and pain is needed for just small things to work.

[dcramer]

I think a fundamental change of perspective is needed. SDK's, APIS's and CLI's are nice, but not what really deliver value. It's the working configuration, that actually ships the correct artifacts to Sentry and makes source-maps from typescript work. And currently Sentry has no solution for this - the only answer is to carefully configure SDK's and CLI usage.

FWIW we are aligned. Sometimes its a tricky balance of us trying to do too much, or accepting a not-quite-mature SDK into the main stream. A good example is our challenge with Next.js support recently. We built out a POC, but even after 2 major iterations we still havent quite nailed the UX.

I think your sentiment aligns well with our improvements to Workflow, but we do have quite a few resources on engineering and so its less of a problem of us providing a curated SDK experience, and more that we need to build better community/feedback cycles to make sure we're doing a great job.

[Kjir]

I have really big issues with the new way the Issues link in the menu works: before I would stay on the project I previously selected, whereas now every time I click on the menu it resets to the first project in the list. I've acted more than once on the wrong project because of this (Alerts suffers the same problem, and I've created alerts in the wrong project more than once).

I would really welcome an improvement here. The same issue happens with the other dropdowns (environments, time range), and those too have caused me more than once to waste time looking at wrong thing (e.g.: an issue that I thought was in production because I previously selected the environment, but it actually only happens in development and the filter was just reset).

[davidenwang]

@Kjir Apologies here on the inconveniences with the new project/env/date selectors. Resetting the value between pages was part of a set of changes to make filtering in the app a little easier to understand, especially for first time users.

To address your issues, we still do support maintaining the selection between pages, similar to how it worked before, by using the "lock" button on the filter dropdown. We have more documentation on the changes here: https://docs.sentry.io/product/sentry-basics/search/#page-filters

Hope this helps!

[Kjir]

That works, it's not intuitive at all, but it does solve my problem, thanks!

[therealarkin]

@davidenwang I've heard a few similar complaints. Do you mind sharing the background of how resets made the filtering easier to understand?

[chadwhitacre]

Internal conversation on this is dialing in on "a typical release workflow and addressing what types of notifications bring the most value to developers who worked on that release." Therefore starting a thread here to collect: what is your typical release workflow? From T-0 = release goes out to prod, what are you doing for the next N minutes/hours/days to understand whether you've effed up? What tools are you looking at? Hopefully Sentry is one of the tools you're looking at. ;) What parts of Sentry are you using? What dashboards and/or notification channels are you depending on? What signals are you looking for, and how are you reacting/adjusting in both normal and abnormal situations? What other stakeholders are involved? What is the most painful part of this process for you right now?

[chadwhitacre]

I'll go first. :]

I don't ship a lot, but when I do my habit is to watch the releases page for 5 or 10 minutes and compare "crash-free rate" with recent historical values for the same. If the values are roughly comparable for 5 minutes or so after reaching 100% adoption then "Phase 0" is done for me and I move on. During that time I do investigate any "new issues" that come up, if any. There aren't that many in the first 10 minutes even with false positives so they are easy to weed out. I haven't shipped a release that broke quickly right away 🤞 but I assume that would pop in the first 10 minutes if it was going to be a big deal.

10 minutes later the next release is probably already going out the door and once that happens I wash my hands of it, take a deep breath, and move on.

Looking at the Release Detail page I am seeing "All Releases" for comparison and the New Issues detail so maybe I will start using this page instead (how long has that been out there? 😬).

[BYK]

Our flow over at @formsort is also similar and I'm actually trying to build an integration connecting GitHub, Jira, Sentry, and Herkou/AWS for full automation and continuous deployment.

Here's what we have/do currently:

1. Every commit on master gets built and creates a "GitHub Deploy" for beta/staging environment
2. The triggered build deploys to AWS & Heroku, creates a Sentry release with a deployment to beta env and runs smoke tests
3. The deployment information also feeds into Jira through their GitHub automation so we can see which issues are deployed to beta
4. At least once a day, we "promote" our beta env to production from Herkou dashboard
5. This promotion causes Heroku to send us a webhook ping connected to Pipedream
6. Over at Pipedream, we do the following:
   1. Create a manual GitHub deployment so this gets synced back to Jira and we can see which issues got deployed to prod
   2. Create a manual Sentry deployment, for prod this time. For some reason Sentry's own Heroku deployment hook doesn't seem to work. Also see Allow specifying projects for release deploys #33648
   3. Send a ping to Slack deployments channel with links to the commit range and links to Sentry release pages for deployed projects.
7. We watch the release pages for ~10 minutes for changes in failure rate, new issues, and crash-free session rate

Ideally, the Slack notification with the list of commits and link to Sentry release page would be done by Sentry itself. I'd also expect Sentry to be smart enough to have a default alert that pings us if the failure rate or crash-free rate changes significantly in T+30 mins after a deploy.

Again, ideally, Sentry would already hook itself into the GitHub deployments API and pick up deployments automatically including env names instead of me deciding where to run the provided GitHub action or decipher its options.

What we are moving to:

1. The manual promotion is done from GitHub Actions with a scheduled event or with a manual workflow_dispatch
2. The actual deploy happens in the same workflow that deploys to beta reacting to the created deployment and protected via environment protection rules
3. Everything that happens in Pipedream earlier now happens inside GitHub actions, including Slack notifications

I think Sentry can save us a huge amount of time and maintenance work by just integrating better with GitHub without requiring a lot of configuration.

[amakhrov]

We have a team of devs shipping code multiple times a day. We have to rely on automation - no way somebody looks at logs upon every release.

Also, for single page apps, there is no strong guarantee it's always the latest code (from the latest release) that the client is running at the moment. Hence, focusing on the latest release only would decrease visibility into new issues.

Alerts for new high volume issues go to a shared slack channel. It would be super nice to be able to @mention a particular slack alias in the notification (so that a person on page duty gets notified)

[Stoff81]

I am not getting what I need out of the Jira Alerts. Its seems that once a issue is created, the alerts don't update the issue if another rule triggers the create again. My use case is as follows:

1. Issue reaches Urgent defect threshold, -> Create Jira alert rule is triggered -> Jira Issue is created with Urgent priority
2. Issue happens more and then reaches Show Stopper threshold -> Create Jira alert rule is triggered -> Pre-existing ticket is not updated to Show Stopper.

[iwithrow]

@Stoff81 I believe not = now, right? Makes sense otherwise.

Curious why you choose Jira severity to drive the notification processes for you? Or atleast it sounds like it, maybe this is just keeping systems sync'd correctly. (I do PM at Sentry, thus I'm asking)

[Stoff81]

Hi,

Not = Not I'm afraid.

I'm not using Jira severity to drive notifications. I am wanting Jira Issues' priority to be updated based on frequency of events in sentry. The only alert which exists for Jira is "Create Issue". If you call this a second time (and the issue already exists in Jira, none of the fields are updated.

I have worked around this by creating a Interal Sentry Integration which connects to a Jira Automation. The Jira automation then calls the sentry issue API using issue_url from the webhook payload, pulls out the jira ID from the response and updates the tickets. Its a lot of pfaffing around, but it does work.

My automation is relying on the annotations field of the issue json response so if that schema changes my rule will break... would be nice for this to just work out the box.