New Feature: High-level changes overview - feedback needed!

[matejak]

Have you ever come across a situation when you wanted to understand how the project changed between two points of time, and git diff was not the best tool you could wish for due to its too fine-grain resolution? And the datastream diff tool's features didn't cover your question?

We have introduced a new experimental metadata in #10761 that could help in this regard. At one point of the build, some of the compiled artifacts are examined by the build-scripts/generate_manifest.py script, and a file build/<product>/manifest-<product>.json is produced.

It is a JSON structure that is very easy to create, and also easy to parse. Although the fact that something is compiled doesn't guarantee its presence in the actual content, it can be a powerful tool to describe changes, and perhaps to provide a stable accompanying metadata format for the content in the future, even if we decide, for some reason, that datastreams are not cool any more.

Right now, the format is not part of any API, it is just connected with a script utils/compare_versions.py introduced in #10768 that can output a diff of two source trees in a human-readable form. That means that it's up to us, the community, to experiment, and agree on the schema of that metadata, so the feature makes most sense to all of us.

Right now, we track:

• rules compiled in the context of a product,
• presence of rule content (e.g. bash, ansible),
• rule platform string, and
• profile composition (rules, values).

So give it a thought, visualize the tree using the jq command, e.g. by running jq . build/example/manifest-example.json and please come up with your opinion regarding what to put there, and conversely what not to. Some data could act as a wearable weight rather than as gold in one's pocket - and we are after the gold.