feat(trufflehog): add link field and deduplicate issues based on it

[brieucR]

Description

Resolving part of #10271

Please note that:

• This solution only works with trufflehog v3 (as trufflehog v2 doesn't come with a link value)
• A migration step is required, to make sure that all existing findings are added with a generated url value (this to avoid all values being marked as duplicates, as hash calculations would be based on empty values).

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Bugfixes should be submitted against the bugfix branch.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
• Model changes must include the necessary migrations in the dojo/db_migrations folder.
• Add applicable tests to the unit tests.
• Add the proper label to categorize your PR.

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	✅	0 findings
Sensitive Files Analyzer	✅	0 findings
AppSec Analyzer	✅	0 findings
Authn/Authz Analyzer	✅	0 findings
Secrets Analyzer	✅	0 findings

Note

🟢 Risk threshold not exceeded.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The code changes in this pull request are focused on enhancing the security and functionality of the DefectDojo application, a popular open-source vulnerability management tool. The changes include updates to the Trufflehog parser, which is responsible for detecting hardcoded secrets and sensitive information in Git repositories, as well as extensive modifications to the Django settings file, which configures various security-related settings, authentication options, and deduplication algorithms for the different vulnerability scanners integrated into the application.

The Trufflehog parser changes improve the reporting capabilities by updating the url field of the Finding object to include a link to the commit where the sensitive information was found. This additional context can be helpful for developers to investigate and remediate the identified security issues.

The changes to the Django settings file cover a wide range of security-related configurations, such as enforcing HTTPS, setting HttpOnly and SameSite flags on cookies, controlling the Cross-Origin Opener Policy, and enabling various authentication options, including social authentication and SAML2. The code also introduces a comprehensive deduplication configuration system and fine-tuning of the hashcode calculation used in the deduplication process, which is crucial for maintaining the accuracy and integrity of the vulnerability data in the application.

Files Changed:

1. dojo/tools/trufflehog/parser.py: The changes in this file update the url field of the Finding object to use the link value instead of the default "N/A". This enhancement provides more detailed information about the location of the detected sensitive information, which can help developers quickly investigate and remediate the issues.

2. dojo/settings/settings.dist.py: This file contains extensive updates to the Django settings, focusing on improving the overall security, authentication options, deduplication configuration, and performance-related aspects of the DefectDojo application. The changes cover a wide range of security-related settings, authentication backends, deduplication algorithms, Celery configuration, logging, and JIRA integration.

Powered by DryRun Security

[kiblik]

As I remember, there was an idea to drop Finding.url (check comment)

django-DefectDojo/dojo/models.py

Lines 2258 to 2262 in f66e6db

	url = models.TextField(null=True,
	blank=True,
	editable=False,
	verbose_name=_('URL'),
	help_text=_("External reference that provides more information about this flaw.")) # not displayed and pretty much the same as references. To remove?

Just wanted to let you know that it might not be the best idea to store this information in the URL field.

[brieucR]

Thanks @kiblik 🙏

I was thinking that

url = link to the flaw

Instead of this, I understand that

url = external source for more resources about the flaw type

I'm a bit confused on what field we should use to better deduplicate findings. Relying on Raw or RawV2 would work for new findings (as proposed in #9899) but I don't know how we could fill the field for the existing issues to avoid all them being set as duplicates because the field is empty.

[kiblik]

I understand your question but not sure I know what would be the best now. I'm sorry.