Find and Report Secrets in Code

• Founder: Abdullah Khawer (LinkedIn: https://www.linkedin.com/in/abdullah-khawer/)

Introduction

This repository has a solution that finds secrets in a git repository using Gitleaks, generates a JSON report based on the findings from Gitleaks by extracting only the relevant information, finds the commit id and commit author for each finding, updates an Atlassian Confluence page with the secrets found based on that generated report and finally sends an alert on Slack.

❓ Where I can run this?

👉🏻 This solution can be executed on any macOS or Linux system either locally or on a remote server. It can also be executed on a CI/CD pipeline.

Usage Notes

Manually on a Local or Remote Server

Prerequisites

Following are the prerequisites to be met once before you begin:

• Following packages are installed on your system:
  • In case of Linux, install the following packages using either ./installation/linux_install_packages.sh script or manually:
    • git
    • jq
    • python3-pip
    • python3
    • make
    • wget
    • golang
    • gitleaks
    • atlassian-python-api
      • Using pip
    • pytz
      • Using pip
  • In case of macOS, install the following packages using either ./installation/macos_install_packages.sh script or manually:
    • git
    • jq
    • python
    • python@3
    • gitleaks
    • atlassian-python-api
      • Using pip
    • pytz
      • Using pip

Execution Instructions

Once all the prerequisites are met, set the following environment variables:

• PATH_TO_GIT_REPO
  • To keep the size of the git repository to be cloned lower to make the job faster (e.g., /Users/Abdullah.Khawer/Desktop/myrepo)
• CONFLUENCE_SITE
  • Atlassian Confluence host link (e.g., https://mydomain.atlassian.net)
• CONFLUENCE_USER_EMAIL_ID
  • Atlassian Confluence user email ID (e.g., myname@mydomain.com)
• CONFLUENCE_USER_TOKEN
  • Atlassian Confluence user token
• CONFLUENCE_PAGE_TITLE
  • Atlassian Confluence page title (e.g., Secrets Detected in the Git Repositories)
• CONFLUENCE_PAGE_SPACE
  • Atlassian Confluence page space (e.g., docs)

And then simply run the following 2 commands in the correct order:

• bash gitleaks.sh
• python3 main.py [TIME ZONE] [REPOSITORY NAME] [BRANCH NAME]
  • Example: python3 main.py Europe/Amsterdam appdev/appdev master
  • Note: Details about supported time zones and their constant names can be found here: pypi.org > project > pytz > Helpers

Automatically via CI/CD Pipeline

Setup Instructions

In order to run it on any GitLab repository, add the following in the .gitlab-ci.yml file that is in the repository:

include:
  - remote: 'https://raw.githubusercontent.com/abdullahkhawer/find-and-report-secrets-in-code/master/ci/.gitlab-ci.yml'

stages:
  - scan

secrets_detection:
  stage: scan
  extends:
    - .find-secrets:scan
  variables:
    CONFLUENCE_SITE: $CONFLUENCE_SITE
    CONFLUENCE_USER_EMAIL_ID: $CONFLUENCE_USER_EMAIL_ID
    CONFLUENCE_USER_TOKEN: $CONFLUENCE_USER_TOKEN
    CONFLUENCE_PAGE_TITLE: $CONFLUENCE_PAGE_TITLE
    CONFLUENCE_PAGE_SPACE: $CONFLUENCE_PAGE_SPACE
  rules:
    - if: $CI_COMMIT_BRANCH == "master"

Where in the rules section, you specify the list of branches to be scanned using if conditions.

The variables are supposed to be created on the repository under CI/CD Settings page.

Notes

• A sample Gitleaks configuration file can be found here if interested in using it: .gitleaks.toml
• The Atlassian user should have access to the Confluence app, the View and Add permissions in the space on it and the Can edit permission on the page in that space. Also, you need to create an API token as the password won't work.

References

A list of useful references can be found below:

• https://gitleaks.io/index.html
• https://docs.gitlab.com/ee/user/application_security/secret_detection/
• https://github.com/gitleaks/gitleaks#configuration
• https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml
• https://docs.python.org/3/
• https://atlassian-python-api.readthedocs.io/
• https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html
• https://docs.gitlab.com/ee/ci/
• https://pypi.org/project/pytz/
• https://git-scm.com/docs/git-blame

Any contributions, improvements and suggestions will be highly appreciated. 😊