Monitoring Your Network Requests

[zoesyc]

Network requests can consume a significant chunk of time in your application and - sometimes - they’re not entirely within your control.

We’ve heard some pretty consistent calls (examples: #50049 and Discord) for some help with monitoring http requests. So we’re currently developing a workflow, like Queries, to help you visualize where time is being spent across your API calls.

Troubleshooting Network Requests

The proposed workflow:

• Starts with an aggregate view of all the network requests across your application - grouped by hosts / services
  
  
  • Key metrics surfaced here are errors (400 and 500 response codes), request duration, throughput (requests per minute), and time spent (in your application) calling each service
    
    
  • (Stretch) See an up-to-date status of the third-party provider - there are some options here and we’d like to hear more about what you’d be interested in seeing (see ‘We want to hear from you!’ section)
    
    
• Once you’ve identified a problematic host, you can get an overview of the endpoints calling this service sorted by time spent
  
  
  • From here, you can immediately get a slow, average, and fast sample for each endpoint
    
    
  • Separately, you can see a distribution of individual samples corresponding to specific response codes within the 200, 400, and 500 classes
    
    
• With each individual sample, you’ll get a complete trace view across frontend and backend services (if both are instrumented) and precise context for where and when the request was made
  
  

This workflow should provide enough signal to quickly see any internal or third-party service outages, regressions, or anomalies negatively impacting your application’s overall performance.

Mockups

1. Network Requests Aggregate View

2. Host / Service Summary View

3. Response Codes Sample View

We want to hear from you!

In addition to Queries, we’re improving Sentry Performance for backend services and we’d like to know learn more about what you’d like to see:

• What kinds of problems with HTTP requests do you see in your applications and how do you debug them?
  
  
• Up-to-date provider statuses: there may be several different options available for how we can track up/downtimes: Using our own system data, monitoring provider status pages, and more.
  
  
  • How would you like to track third-party provider status?
    
    
• Are you currently using GraphQL? What is your experience with GraphQL in Sentry?
  
  
• We’re also looking into building additional workflows for Caches and Queues! What are some key metrics for these workflows?
  
  

Feel free to share your thoughts in the discussion below - or you can find us on Discord and X.

[MisterJimson]

This looks really nice.

I would want to make sure arbitrary status codes can be filtered on, instead of preset 2xx, 4xx, 5xx. You never know how an application you don't control will behave, and sometimes digging into a specific status code is needed.

2xx, 4xx, and 5xx are solid defaults.

[zoesyc]

Thanks, @MisterJimson!

To clarify - are you referring to filtering on the Response Codes Sample view (#3) or earlier in the workflow?

[MisterJimson]

Yes for mockup 3!

[shaedrich]

It would be nice to have something like the cron job monitoring health overview to see which APIs/endpoints are healthy and which are not, both currently and by percentage/average:

It would also help if endpoints could be grouped by API/service/host as well as ungrouped in the overview.

Since you already link to the status page in general,

if you get the necessary information from there, you could display if the 5xx is related to a reported outage, similar to how you flag bad releases and suspect commits.

[zoesyc]

Thanks for the feedback @shaedrich!

Was looking to clarify a few things to make sure I understand:

It would be nice to have something like the cron job monitoring health overview to see which APIs/endpoints are healthy and which are not, both currently and by percentage/average

What do you mean by percentage and/or average? Is this referring to duration? or healthy vs. failed events?

It would also help if endpoints could be grouped by API/service/host as well as ungrouped in the overview.

• In the mockups, the endpoints are grouped by API/ service/ host in the Aggregate view (small fixes to get example_project working #1) - did you mean that you would like to see them grouped elsewhere?
• By 'ungrouped in the overview', are you looking to see the exact endpoints you're calling within a host within the host summary?

[shaedrich]

What do you mean by percentage and/or average? Is this referring to duration? or healthy vs. failed events?

I meant "healthy vs. failed events"

grouped by API/service/host

In the mockups, the endpoints are grouped by API/ service/ host in the Aggregate view

Yes, but it looks like currently they are just one of these.

ungrouped in the overview

By 'ungrouped in the overview', are you looking to see the exact endpoints you're calling within a host within the host summary?

Let me clarify these two with one example:

Ungrouped (by request)

2024-03-06T08:15:01Z GET    https://example.com/api/users    users.all    200
2024-03-06T08:15:22Z GET    https://example.com/api/groups   groups.all   400
2024-03-06T08:15:28Z POST   https://rest.sample.com/users    users.create 201
2024-03-06T08:15:53Z PUT    https://graphql.sample.com/users users.update 204
2024-03-06T08:16:04Z DELETE https://example.com/api/users    users.delete 503
2024-03-06T08:16:11Z GET    https://example.com/api/users    users.all    501

Grouped

by host

2024-03-06T08:15:01Z – 2024-03-06T08:16:04Z example.com (4)
2024-03-06T08:15:28Z – 2024-03-06T08:15:53Z sample.com (2)

by service

2024-03-06T08:15:01Z – 2024-03-06T08:16:04Z example.com/api (4)
2024-03-06T08:15:28Z                        rest.sample.com (1) <-- here
2024-03-06T08:15:28Z                        graphql.sample.com (1) <-- here

by endpoint

2024-03-06T08:15:01Z – 2024-03-06T08:16:04Z example.com/api/users (3)
2024-03-06T08:15:22Z                        example.com/api/groups (1) <-- here
2024-03-06T08:15:28Z                        rest.sample.com/users (1)
2024-03-06T08:15:28Z                        graphql.sample.com/users (1)

by action

2024-03-06T08:15:01Z – 2024-03-06T08:16:04Z GET    users.all    example.com/api/users (2)
2024-03-06T08:15:22Z                        GET    groups.all   example.com/api/groups (1)
2024-03-06T08:16:04Z                        DELETE users.delete example.com/api/users (1)  <-- here
2024-03-06T08:15:28Z                        POST   users.create rest.sample.com/users (1)
2024-03-06T08:15:28Z                        PUT    users.update graphql.sample.com/users (1)

[pats]

Great concept!

Few suggestions:

• maybe some kind of percentile matrix instead of avg values?
• what about 'RED metrics' ? devops culture has some "best of" standards

[jeengbe]

This is amazing. It's the final piece Sentry is missing to replace other observability platforms, as we still cover uninstrumented services this way.

Few suggestions:

• There's perhaps a possibility to somewhat merge this with the Queries view (or embed a filtered Queries view in this) such that requests to databases also show up in this list.
• The *.host.tld is a bit problematic. We reach most of our services with service-a.host.tld, service-b.host.tld, etc. and currently, they're all grouped as *.host.tld.
• It would be very nice if this could somehow integrate data from metrics instead of processing transactions. With a low transaction sampling count, data points are rather sparse (especially for infrequently visited routes). Our "request per minute" count is currently very off. If, instead (or additionally), SDKs would emit metrics for any instrumented (but not sampled) network call, both parties would likely benefit from that (as I guess metrics are easier to store/process). Transactions could complement metics-based data such that linked transactions are displayed if available.