SPDX File Information

[fdegir]

SIG Interoperability has been having discussions around a standardised approach to handling metadata that are produced and consumed by CI/CD pipelines. As part of these conversations, contributors identified few key areas to focus on such as commit sha and artifact metadata.

Some of the contributors highlighted the existing work done by SPDX community and the SIG participants reached a consensus to start exploring synergies to collaborate with SPDX so we can look at what is currently available, what is missing, overlaps, and gaps between what SPDX community did and what CI/CD communities deem necessary.

The idea with this discussion is to start looking into SPDX Spec and more specifically to File Information from the spec to see how well it covers the needs of CI/CD domain and contribute to SPDX accordingly.

Tagging few people who's shown interest in this topic - @sbtaylor15, @jstrachan, @kestewart, @tracymiranda

[fdegir]

Cross-referencing the topic from SIG Events so folks can check what is happening in both SIGs: cdfoundation/sig-events#50

[mattyn]

The SPDX File Information spec covers some important information, particularly related to copyright and licensing. It also covers basic file information like name, type, and checksum, although the file type options seem limited. For example, it might be valuable to represent file type in terms of the role of a file in a package in addition to its format (source code vs. configuration, or test vs. runtime code). It might also be valuable to represent languages and specific file formats (Java vs. Go, or GZip-compressed tar vs. ZIP).

I think there are additional categories of file-related data that would be important in the CI/CD domain. One category could cover fine-grained version control metadata, like whether a file was merged or directly edited, the SCM revision ID of the change that last modified the file, or an SCM-specific identifier for the file version (e.g. the blob ID in git). Another category could cover information about file contents and changes; for example, file size, encoding, or lines added/removed/changed. It might also be interesting to represent relationships to other versions of the same file that were stored in different locations (renames, or files copied from other packages).

However, when I read the rationale for SPDX, I'm not sure if all this information is relevant to the purpose they outlined; it seems focused on identifying which open source software is used and which licenses govern its use.

[kestewart]

SPDX is for "software", not just open source. It started in open source compliance 10 years ago, but has evolved to handle many other use cases since then, and is continuing to evolve.

The file type options were based on the standard mime types - and if we can describe how it will be used, adding role, language & file format info can certainly be considered. File size should be straight forward, and is already implied in the snippet type.

I'm wondering if lines aded/removed/etc. can leverage snippet.

In terms of relationships between files over time, there is a rich set of relationships already defined. See: https://spdx.github.io/spdx-spec/7-relationships-between-SPDX-elements/

We're working on the next rev of SPDX where we plan on including "build" data (aka pedigree) as an optional profile. I'd very much welcome a discussion as to the specific types of fields you want to see captured, and we can work up a proposal for the upcoming rev of the specification. There are a couple of other folks interested in fleshing out this profile as well and it might be useful to meet and discuss what's possible and useful.

I've started a doc at: https://docs.google.com/document/d/1BYSw2bbCsLvt4nsMt3AHpF_H36dRFLJwQb3d93P1xNE/edit, and have added the fields you ref above that aren't in the spec already. Please feel free to add any fields you're thinking will be useful, and we can iterate there a bit, then call a meeting, and then work up a proper pull request for profile.

[mattyn]

Thanks for clarifying the intent of the standard. I'll take a look at the doc and add some ideas.

[fdegir]

hi @kestewart
A question came up while we were discussing SPDX about the possibility of extending SPDX.

Is it possible to extend SPDX with custom and potentially proprietary metadata?

If so, can you please pass the link to where this is documented?

[justinabrahms]

I've been looking into SBOMs recently and SPDX is often mentioned alongside CycloneDX. Just adding it here in case others haven't seen it. https://cyclonedx.org