Nosey Parker v0.17.0

[bradlarsen]

NOTE: Prebuilt releases take a bit to create and will be added soon.

Additions

• A new --ignore-certs command-line option has been added to the scan and github commands. This option causes TLS certificate validation to be skipped (#125; thank you @seqre).

• The scan and github commands now support the --all-organizations flag. When supplied along with a custom GitHub API URL, Nosey Parker will scan the provided GitHub instance for all organizations to be further enumerated for additional repositories (#126; thank you @seqre).

• New rules have been added (thank you @gemesa):

  • Adafruit IO Key (#114)
  • Blynk Device Access Token (#117)
  • Blynk Organization Access Token (URL first) (#117)
  • Blynk Organization Access Token (URL last) (#117)
  • Blynk Organization Client ID (URL first) (#117)
  • Blynk Organization Client ID (URL last) (#117)
  • Blynk Organization Client Secret (URL first) (#117)
  • Blynk Organization Client Secret (URL last) (#117)
  • Docker Hub Personal Access Token (#108)
  • Doppler CLI Token (#111)
  • Doppler Personal Token (#111)
  • Doppler Service Token (#111)
  • Doppler Service Account Token (#111)
  • Doppler SCIM Token (#111)
  • Doppler Audit Token (#111)
  • Dropbox Access Token (#106)
  • particle.io Access Token (URL first) (#113)
  • particle.io Access Token (URL last) (#113)
  • ThingsBoard Access Token (#112)
  • ThingsBoard Provision Device Key (#112)
  • ThingsBoard Provision Device Secret (#112)
  • TrueNAS API Key (WebSocket) (#110)
  • TrueNAS API Key (REST API) (#110)
  • WireGuard Private Key (#104)
  • WireGuard Preshared Key (#104)

• A new generate command has been added, which generates various assets that are included in prebuilt releases:

  • Shell completion scripts via generate shell-completions
  • A JSON Schema for the report -f json output via generate json-schema (#128)
  • Manpages via generate manpages (#88)

Fixes

• Several rules have been fixed that in certain circumstances would fail to match and produce a runtime error message:

  • Google API Key
  • ODBC Connection String
  • Sauce Token

• The netrc Credentials rule has been modified to avoid a runtime message about an empty capture group.

• The JSON Web Token (base64url-encoded) rule has been improved to reduce false positives. Thank you @saullocarvalho for the bug report.

• The prebuilt releases now include shell completion scripts for bash, fish, elvish, powershell, and zsh, instead of 5 copies of the zsh completions (#132; thank you @Marcool04).

Changes

• The minimum supported Rust version has been changed from 1.70 to 1.76.

• The data model and datastore have been significantly overhauled:

  • The rules used during scanning are now explicitly recorded in the datastore. Each rule is additionally accompanied by a content-based identifier that uniquely identifies the rule based on its pattern.

  • Each match is now associated with the rule that produced it, rather than just the rule's name (which can change as rules are modified).

  • Each match is now assigned a unique content-based identifier.

  • Findings (i.e., groups of matches with the same capture groups, produced by the same rule) are now represented explicitly in the datastore. Each finding is assigned a unique content-based identifier.

  • Now, each time a rule matches, a single match object is produced. Each match in the datastore is now associated with an array of capture groups. Previously, a rule whose pattern had multiple capture groups would produce one match object for each group, with each one being associated with a single capture group.

  • Provenance metadata for blobs is recorded in a much simpler way than before. The new representation explicitly records file and git-based provenance, but also adds explicit support for extensible provenance. This change will make it possible in the future to have Nosey Parker scan and usefully report blobs produced by custom input data enumerators (e.g., a Python script that lists files from the Common Crawl WARC files).

  • Scores are now associated with matches instead of findings.

  • Comments can now be associated with both matches and findings, instead of just findings.

• The JSON and JSONL report formats have changed. These will stabilize in a future release (#101).

  • The matching_input field for matches has been removed and replaced with a new groups field, which contains an array of base64-encoded bytestrings.

  • Each match now includes additional rule_text_id, rule_structural_id, and structural_id fields.

  • The provenance field of each match is now slightly different.

• Schema migration of older Nosey Parker datastores is no longer performed. Previously, this would automatically and silently be done when opening a datastore from an older version. Explicit support for datastore migration may be added back in a future release.

• The shell-completions command has been moved from the top level to a subcommand of generate.

---

This discussion was created from the release Nosey Parker v0.17.0.

[bradlarsen]

I just updated the macOS release artifacts. The original artifacts were not codesigned at all; now they are codesigned and notarized.

[jonstewart]

Dumb question: is it feasible to np for Windows, outside of the docker container?

[seqre]

That's not a dumb question! Some initial research about it has been done fairly recently (see #121), but it's still in the works.

[bradlarsen]

You are not the only person who has asked for this, @jonstewart. It's a reasonable question.

Your best existing option on Windows is to use the Docker image.

To further clarify: native Windows support for Nosey Parker should work in theory, but some porting / testing / CI effort is required. Though I'd like the project to work on Windows, getting it there is a low priority for me right now.

Another option might be using WSL2, but can't comment meaningfully on that option since I've never tried it.

[jonstewart]

Thank you for prompt replies! I may have a team member who's been cursed with black magic cross-compiler knowledge to take a look at it next week.

[bradlarsen]

I'm happy to answer any questions that pop up. I started writing down some notes in #121.