Feedback for Sentry+Codecov

[Dhrumil-Sentry]

Hi Everyone!

As you may have heard, Codecov has recently joined the Sentry family!

Sentry and Codecov aim to give developers shared insights about error monitoring and test coverage so that they can ship more robust code.

Our teams are considering some of the following ideas for a Sentry-Codecov integration

1. Showing information about test coverage in a Sentry issue’s stack trace so that you know if the error code is partially or fully covered by tests

2. Showing Sentry issues related to particular commits and code files in Codecov

3. Showing error and test coverage information related to files being changed in pull requests in GitHub

We’d love to hear your thoughts about what you would like to see in a Sentry-Codecov integration. Let us know 👇

[MartinMarx]

Hey @Dhrumil-Sentry

As a Sentry + Codecov user, I'm very excited about this announcement.

The 3 ideas your mentioned are a good start. I would also suggest to add coverage information on each release so we can track our code coverage progress directly on Sentry (reach graph, diff with previous release, impacted files). Maybe we could also imagine to add some alerts, for example when the coverage percentage is below a certain threshold?

Anyway I can't wait to see where it is going, well done!

[therealarkin]

@MartinMarx what are some useful thresholds? and why?

[kavdev]

The threshold would have to be able to be set in project settings. Each org has their own standards

[zpg6]

Congrats on the announcement! I think this could prove to provide developers with more tools to make sure errors caught won't happen again. Some ideas:

1. Mark an error in a special way if it results from a function that has full coverage, we all know coverage doesn't insist all edge cases are covered. Somewhat of a "blindspot detection".

2. Helping prioritize test cases for code that has most errors related to it.

3. This one might be a stretch, but somehow listening to a failing (pure) function's inputs and enabling them to be extracted to a unit test case. This can be such a manual process and could really improve code quality if it were more automated. Even starting with only covering a few languages and just suggesting snippet for test runners, kind of like how REST clients provide snippets for running a request in your language of choice. Could be taken as far as a dependabot-like app that makes a PR with additional test cases added.

[Dhrumil-Sentry]

@zpg6 Thanks a lot for the feedback!

• We 100% agree that showing partial/fully covered status for error lines is helpful and our first iteration will show this info

Wrt helping you prioritize test cases for code with the most errors- Would you like to see those insights in Sentry or your Codecov app?

[zpg6]

I would like to see this on codecov where it shows source code and maybe even in PR comments.