Codecov Integration for Sentry #43826
Replies: 6 comments 17 replies
-
|
The biggest challenge I see today is the complexity in setting it up. We need to find a much easier way to get folks access to this feature, ideally without connecting to GitHub inside of Sentry. |
Beta Was this translation helpful? Give feedback.
-
|
@dcramer code mappings help us derive in which repo the file will be found. If there are no code mappingswe could determine it on the fly (not completed work), however, all of this depends on the customer giving us permission to their repositories (via Github, Gitlab, Bitbucket). If we do not have permission, we can only query public repos and with a much lower API rate limit (the rate limit via our Github Apps gives us higher rates). Also note, that Codecov has a 1 to 1 match to the customers' repositories (getsentry/sentry -> gh/getsentry/sentry). I believe the way forward is to create more code mappings. We have some ways to create a lot more (besides adding more integrations and languages). |
Beta Was this translation helpful? Give feedback.
-
|
I guess, my point is cant you achieve this without such a complicated approach? we have filename and abs_path in the SDK just to try and normalize stacks to the repo. Like generally speaking the pattern is a one-time-thing in each project and you can map everything from there e.g. if the source code is in lib/foo/bar but the stack says foo/bar we certainly can figure that out somehow.. or make a dirt cheap config that doesnt require a complicated repo crawling thing to achieve or even more so, couldnt we put the code mapping stuff on the codecov side somehow given they already have repo access? |
Beta Was this translation helpful? Give feedback.
-
|
f/u from Slack - two things identified:
|
Beta Was this translation helpful? Give feedback.
-
|
Q: Can we enable this feature for a Sentry+Codecov user without them having to install Sentry's GH App Integration? @dcramer has surfaced the two most relevant problems that are currently solved by a github integration on the Sentry side:
The code mapping part is actually easier. In reply to:
Codecov actually already has this functionality, because we've needed it for years to map the paths in coverage reports generated in a CI environment to their corresponding paths in a repository. We call these Additionally, we apply the same concept to path fixes in production environments to power our Impact Analysis feature. This isn't well documented (easy to fix), but here's an example of applying them in a Laravel project: https://docs.codecov.com/docs/impact-analysis-quickstart-laravel#troubleshooting Personally I don't think this is as slick as deducing code mapping automatically since it still requires users to write the path fix, but it is a Codecov facing solution to the code mapping problem that exists today. We could apply those for the Sentry integration and then Sentry could send over the production paths and rely on Codecov to do the mapping.
Determining org name and repo name is fairly straightforward to solve poorly: just ask users to provide those things in Sentry's settings (e.g., in form fields in the settings page for a project), and you won't need the GH integration. That's pretty gross and potentially error prone, though. You also don't have a great way, that I'm aware of, to validate that the user-supplied org name and repo name is correct. So of the two problems @dcramer surfaces, I actually think determining the org name and repo name reliably and correctly without relying on human input is actually the harder one. |
Beta Was this translation helpful? Give feedback.
-
|
@eliatcodecov that harder one is what the code mappings derivation system I designed solves (not deriving the org part). It could be solved with Github's search API with enough unique file paths. |
Beta Was this translation helpful? Give feedback.
-
|
OOC (apologies if this is the wrong place to ask) — is there some other pre-req or flag for access? I'm admin for my own workspace and wanted to give this a whirl but am not seeing that toggle:
(I do have GitHub enabled for the relevant repository and a concomitant Codecov account, but maybe there's some linkage somewhere that's missing?) |
Beta Was this translation helpful? Give feedback.
-
|
Hey, @snigdhas @Dhrumil-Sentry can you add |
Beta Was this translation helpful? Give feedback.
-
|
Hi @lorn, I've added your org to the beta |
Beta Was this translation helpful? Give feedback.
-
|
Hi @snigdhas, @Dhrumil-Sentry, can you please add my org as well? It's BePark on GitHub, and izix on Sentry, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @AdamiecRadek, I've added |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Hi All, The Codecov integration for Sentry is now generally available! We've also launched a new Codecov promo pricing for new and existing Sentry customers that leads to ~50% savings for 5 Codecov Pro seats. You can read more about this here We're also doing a live AMA session with Sentry and Codecov and folks to answer any questions you may have about Codecov and this integration. |
Beta Was this translation helpful? Give feedback.
-
|
I understand, your purchase of Codecov was for you to make money off of us. But isn't having the dependency of Codecov AND GitHub a bit propritary and limiting? I think, it would be easier to just upload a cobertura file? |
Beta Was this translation helpful? Give feedback.
-
|
Codecov is what allows you to upload said format. Ultimately we’d have to
build (and pay for) infrastructure to process coverage in a variety of
formats and conditions (it’s not as easy as just handling a single file
uploaded). It doesn’t make sense for us (financially, if nothing else) to
repeat that process and the ultimate value add here is for people who are
focused on test and prod (via both our products).
On Thu, Mar 30, 2023 at 12:38 AM Sebastian Hädrich ***@***.***> wrote:
I understand, your purchase of Codecov was for you to make money off of
us. But isn't having the dependency of Codecov AND GitHub a bit propritary
and limiting? I think, it would be easier to just upload a cobertura file?
—
Reply to this email directly, view it on GitHub
<#43826 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAFYOUXP5USRQBACMJXH6LW6UZ5NANCNFSM6AAAAAAUKXG364>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Sent from my phone. Pardon the typos.
|
Beta Was this translation helpful? Give feedback.
-
|
One thing we're running into with Sentry and Github, is that for security reasons we don't want to give you read access to source code (nor write access to repo hooks). Ideally a Codecov integration would use Github Actions (where we can run your CLI, pinned via SHA, for security) and push the coverage reports or other data to your API. That way, you don't need source code access, and your CLI (via Github actions) would be the tool that sends data, update PRs with comments, etc. It's a much more secure design. |
Beta Was this translation helpful? Give feedback.
-
When you triage a bug do you question if the code was tested and if better tests could have prevented this bug in the first place? We asked ourselves this same question and that led us to build an integration with Codecov (Now a part of Sentry!).
The Codecov integration for Sentry will let you see the untested code or partially or fully covered code that’s causing errors directly in your issue’s stack trace, allowing you to avoid similar errors from happening again.
Getting Started
Get the Owner or Manager of your Sentry account to:
Additional Requirements
You just need to have the GitHub integration for Sentry installed and have your test coverage data in Codecov to start using this integration. You also need Code Mappings to be set up for your GitHub integration- We need this to identify the source code file for showing you test coverage!
Don’t use GitHub? Sit tight, GitLab and Bitbucket support is on the way!
New to Codecov?
New to code coverage? No problem. Signup for Codecov and get started for free.
We want to hear from you
The Codecov integration for Sentry is now generally available for all our customers.
Please comment below to tell us about any bugs & issues or to give us feedback on what else you’d like to see in this integration. You can also email us at codecov-integration@sentry.io
Beta Was this translation helpful? Give feedback.
All reactions