Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Initial Draft #1

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

RFC: Initial Draft #1

wants to merge 2 commits into from

Conversation

captn3m0
Copy link
Member

@captn3m0 captn3m0 commented Dec 17, 2022

Context

I've mentioned a plan for releases.json in a few comments in the past:

The intent has been for us to publish a specification to closely match our API, that can then be used by vendors to publish their own data in a machine-readable format. For most products, they can just set a redirect from their website to endoflife.date API till stay start publishing this.

RFC

This is a v0-draft, requesting feedback and comments. I've tried to keep our recommendations in mind, with the caveat that this is meant to be consumed by code - so we can be verbose.

Left a lot of TODOs in the specification, and haven't filled it around properly (missing values etc), but the intent should be clear for all fields.

Extensions:

Down the line, I can expect some extensions to make this more useful:

  1. Embedding external URLs for specifying each product, instead of including all on the same page. That way a vendor like Microsoft can have a single well-known URI for releases.json, but have this published over a dozen files across various teams.
  2. Specification of Source Code Configuration
  3. PURL/CPE Linkages.
  4. BOM/SBOM Linkage for a version.

Alternatives

The current draft uses a top level releaseCycles key to keep the specification brief and easy to use. However, this might make it complicated to generate. I've kept only the supportedReleases field mandatory to promote publishing anything over nothing.

But an alternative would have been for this file to be more exhaustive and cover each version individually:

versions: [
{
  id: 5.2.3
  eol: 2022-12-14
  support: 2022-12-01
  link: 'https://example.com',
  lts: false
  identifiers: // List of PURL identifiers
}
]

However, this might become extremely verbose, especially for products that have lots of versions. Release Cycles are a common enough abstraction that should apply to most cases. The advantage with this approach would have been the possible exhaustive list of versions, which would result in better matches against PURLs for eg.

@captn3m0 captn3m0 added the documentation Improvements or additions to documentation label Dec 17, 2022
@captn3m0 captn3m0 self-assigned this Dec 17, 2022
## Introduction

This specification documents the `releases.json` file format, which is intended to be used by vendors to publish
information about various releases of the artifacts produced by them.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

produced by them -> they produce


**Product**: is an artifact produced by the vendor, typically a software or device.

**Release**: is a singular release made for a product. It is identified by its version string, which is specified without any `v` prefix.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a singular -> a single


**Product**: is an artifact produced by the vendor, typically a software or device.

**Release**: is a singular release made for a product. It is identified by its version string, which is specified without any `v` prefix.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without any v prefix -> without any prefix (such as v) or suffix (such as -final).


**End-of-Life**: When a product reaches End-of-Life, no further updates are expected, including security fixes.

**Commercial Support**: Expectation of receiving fixes from a vendor on payment to a vendor - not necessarily first-party.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Difficult to understand. Maybe something like:

expectation of receiving fixes in return for payment - not necessarily from the original vendor.


**Commercial Support**: Expectation of receiving fixes from a vendor on payment to a vendor - not necessarily first-party.

**Long-Term-Support**: A Long-Term-Support release cycle is expected to be maintained for a longer time period than usual releases.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Long-Term-Support" -> "Long-Term Support".

releaseCycles: [
{
// Release Cycle Identifier
id: "5.2",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name is not enough ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name should be human readable (similar to how we use releaseLabel), while id should be friendly to version matching.

// via a primary non-archive source.
// For devices, this would imply the device is available in the market
// for software products, they should be downloadable on the primary source.
availability: true/false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discontinued may be more explicit.

// TODO: Find a better name
// Whether active development is happening in this release cycle.
// Use date for the last date of activeDevelopment.
development: true/false/date
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it the current endoflife support field ? If no I think we should add the support field too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the current eol support field, but renamed to development for clarity (This release cycle sees active development - new features, bug fixes etc)


// Support Variants.
// TODO: Find better name, maybe "supportVariants"?
plans: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an array ? Is there a lot of products having multiple support plans ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should drop eol/lts etc from the releaseCycle, and only keep it inside plans. And specify a default plan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes thing more complex IMHO. Do you think it is frequent to have multiple "plans" for the same product ? Maybe a concrete example would help us see things more clearly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping one top-level object for date/EOL data, and multiple sub-level ones will only make matching more complex. Having an array of plans will be more explicit, and easier for matching by scanners and others.

Array: Used a dictionary here to provide an identifier for each plan. But can add a id field instead.

development: true/false/date
}
]
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an idea : we could group dates together, for example :

"development": {
  "active": true/false/date
  "support": true/false/date
  "security": true/false/date
  "eol": true/false/date
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't get this one, why are we grouping this under development?

// This must match one of the latestVersion
// TODO: Decide between supportedVersions or supportedReleases
// Mandatory
supportedReleases: [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If think we should only have releaseCycles and make most of the fields in releaseCycles optional.

This way the vendor could only say it supports some cycle without specifying the version (which is even lighter / less work for him). For example:

releaseCycles: [
    {
      "id": "5.2",
      "supported": true
    }
]

IMHO endoflife.date is more about release cycles than individual versions (at least in its current form).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This moves the burden of "whether a version is supported or not" to the scanner, and is highly dependent on how you do matching - it will be prone to bugs and confusion. Using supportedReleases shifts that back to the vendor, by being explicit about the latest and supported releases.

endoflife.date is definitely about releaseCycles, but not every software fits neatly into it (see windows versions for eg). SupportedReleases is something that fits everywhere, and is much simpler to use - both for publishers as well as consumers.

While endoflife.date project will benefit from releaseCycles, that is not something the ecosystem really asks for - we're asking for it to make our lives easier. What the world needs is "is my version in a list of supported versions".

@captn3m0
Copy link
Member Author

captn3m0 commented Jan 5, 2023

Was looking at @witchcraze's EOL-Dataset: https://github.com/witchcraze/EOL-DataSet, which is trying something similar.

notable differences:

  1. Multiple source URLs
  2. Type, and provider for each plan
  3. TBD declaration for dates, with approximate markers ("TBD(~2022)")
  4. Each support plan can include Distro-specific EOL Dates. (see https://github.com/witchcraze/EOL-DataSet/blob/master/Lang/Lang_PHP.json for eg).
  • 1 and 2 are probably good additions.
  • 3 is tough to add in a specification like ours, but maybe we should support YYYY as a valid ISO year, or YYYY-MM as well for approximations?
  • 4 is something I think is better handled by scanners and tooling. releases.json should be published separately by the PHP Project (for php itself), and redhat for the rhel releases. A scanner can rely on the latter data to provide accurate information.

It still has some of our shortcomings:

  1. No way to differentiate between levels of support (such as only certain repos being supported, or certain packages)
  2. Freemium models of support (such as free Ubuntu Pro subscriptions, with limits)

👋🏽 @witchcraze: It will be nice to hear your thoughts on this draft as well.

@witchcraze
Copy link

witchcraze commented Jan 13, 2023

Thank you for your mention, and sorry, I missed this.
Honestly I was not able to track recent speedy update of this project.
I'll read related posts and documents, and make comment.

@witchcraze
Copy link

For now, let me comment about witchcraze/EOL-DataSet.

Purpose of this project was rounghly notice for me of changing EOL of Software Collectino and AppStream of RedHat.
Since they offten updated date with no announce, and sometimes removed and re-added recode.


3 is tough to add in a specification like ours, but maybe we should support YYYY as a valid ISO year, or YYYY-MM as well for approximations?

As you know, there are many version without EOL.
If how to show such a status is defined, I think it is no problem. Blank, remove field will be ok.

Personally I do not like YYYY EOL and monthly range EOL.
Maybe YYYY-MM should be allowed, since YYYY-MM is widely used for EOL.

Sometimes, this project (and I) use forecast EOL.
For example, EOL of ruby 2.7 will be 2023-04-12, but this is not shown on official site.
To show data strictly, status of EOL may be required. (I think it will be too redundant ...)
"officially fixed", "calucurated by policy", "forecast by trend from previous"

4 is something I think is better handled by scanners and tooling.

Yes, strongly agree.
In fact, maintaining how to match EOl of non-upstream products are time consuming task.

As I wrote before, I think this is convenient in some cases.
In the company, I made such a comparable timeline slides focused several major products, and it is one of popular documents.
Upstream and major OS packages are written with each version and EOL in each product pages.
Even if sometimes OS vendors will not provide security patch which is provided by upstream, many users selected long term supported OS packages in our organization.

As I wrote before, I felt this project focus only upstream EOL, but if this project covers major downstream EOL, it will expand user's opinion.


I'll try check related thred and Initial Draft later

@witchcraze
Copy link

witchcraze commented Jan 14, 2023

@captn3m0

Sorry for my late. Let me write comment for draft.
I think this project is making RFC which shows support, especially eol information. If I have any misunderstanding, please let me know.
// As I'm not core member of this project, I'll not use review function in this time.

Terms

End-of-Life: When a product reaches End-of-Life, no further updates are expected, including security fixes.

I felt this explanation is good for many cases.
But if this project will cover paid supports and device(server hardware?), more terms and life cycle phase will be required, I think.

Someone will use EOL as end of software update.
Someone will use EOL as end of user support.
Someone will use EOL as same meaning of end of sales.
Someone will use EOL as same meaning of end of providing hardware parts.

EOL of OS/firmware update may come before EOL of providing hardware parts.
Sometimes later EOL will be more important.

If too much redundancy is allowed, I'll propose child items under eol.
This will help user to understand support status.

        eol : [
            sales : YYYY-MM-DD,
            feature update : YYYY-MM-DD,
            security patch : YYYY-MM-DD,
            hardware repair : YYYY-MM-DD,
            customer support : YYYY-MM-DD,

or

        eol : [
            [
                type : feature update
                date : YYYY-MM-DD
            ],
            [
                type : security patch
                date : YYYY-MM-DD
            ],

In this case, usable type is defined, and type will increase with schema version

Support: The expectation of receiving timely bug fixes for any defects in the product.
Commercial Support: Expectation of receiving fixes from a vendor on payment to a vendor - not necessarily first-party.

Support has many meaning like EOL.
If only Support is written, someone and I think it will include feature update, security patch and inquiry support, (and some cases ,emergency incident support).
How about adding Supported contents is defined by each product and support type.

As you commented, I think Commercial Support is type of support, and upstream OSS support, 3rd party paid support and agency support are also support type, I think

I'll propose adding Support item and moving policyURL and supportedReleases under Support.
Usable type is defined, and type will increase with schema version

This can show fork distributions as 1 product data. (OpenJDK, K8S, maybe RHEL, and so on)
This can show each channel release of some products. (channel → support type)
This can show EUS of RHEL too, if someone want to show it.

To cover explanation of supported scope, I'll propose using policyURL.
This will help developpers to design scanner or matching script.
Showing those scope in API will be complicated, and machine will not able to solve them all.

  releaseCycles: [
    {
        [
            Support : [
                type: community support,
                provider: upstream community,
                name: community support,
                policyURL : [
                    https://****
                ],
                supportedReleases: [
                    "5"
                ],
            ],
            releaseCycles: [
            ]
        ],
        [
            Support : [
                type: paid support,
                provider: Oracle,
                name: Premier Support,
                policyURL : [
                    https://****
                ],
                supportedReleases: [
                    "5",
                    "4"
                ],
            ],
            releaseCycles: [
            ]
        ],
        [
            Support : [
                type: paid support,
                provider: Oracle,
                name: Extended Support,
                policyURL : [
                    https://****
                ],
                supportedReleases: [
                    "5",
                    "4",
                    "3"
                ],
            ],
            releaseCycles: [
            ]
        ],

Specification

Maybe definition of product is required.
Ideally these "Product Name" can matche one of item in this project or general common understanding.

For example, elastic suites.
I agree separating pages too.
If final output is {hostname}, {product name}, {version}, {EOL Date} and product name is each software, it will be easier to understand for human and machine.

Keys

For hardware, product, model, version, option(e.g. intel type. AMD type) maybe required to identify those eol, but I do not know detail.
Comment from hardware expert is needed.

Product Specification

url: ""

To check EOL and vulnerability, these url type will be useful.
Of course, these information will be redundant to provide EOL.

  • product outline
  • Release note
  • Change log
  • Development (e.g. github project)
  • Community (forum, ML archive, Slack)
  • Announcement (News, Blog including release or security information)

supportedReleases

If child items under eol I proposed above is used, I think you can remove this item.
User will judge their supportedReleases by child items.

I felt these versions would match id in releaseCycles by name - supportedReleases.
For example, supportedReleases os PHP is ["8.0", "8.1", "8.2"].

But this shows latestVersion.
How about using latestSupportedVersions instead.
Anyway, need to explain the contents of this support.

@captn3m0
Copy link
Member Author

I think this project is making RFC which shows support, especially eol information. If I have any misunderstanding, please let me know.

Yes, that's the intent of the specification - provide a clear and easily published (static, not dynamic) machine readable document with product support information.

Noting down issues and possible solutions:

Dates

Switching to ISO8601 makes sense. It supports Dates/Datetimes/Approximate Dates (YYYY-MM) and even ranges (that we should probably not support)

Dealing with non-upstream products

I agree this is worth solving, and there is some consensus on this being downstream. If this specification works, we can use the releases.json from downstream vendors (say RedHat) to augment the scanners easily.

Meaning of Support/EOL/...

Clear problem - I think having sub-definitions (say like active-development, ticket-accepted, security-patches-available, patches-on-request), and then defining each plan as providing a subset of these is probably the cleanest approach.

However, this then makes it much harder for third-parties to consume this data - there's no easy shortcuts, like picking eol or support dates as granted.

Supporting forks/channel/downstream projects in same data

I would mark this as a non-goal. For eg, the OpenJDJ/Oracle Java/Azure JDK etc projects should all publish their own releases.json file with separate product identifiers.

While other projects (such as endoflife.date) could collate this information and publish them together, and the specification already supports multiple products (products, the top-level key is an array) - I'd like to avoid clubbing such variants under the same product-id.

Unsure about providing plan level URLs, but don't see the harm in it.

Defining Product

Good point

Separate links to various pages

Links often to go official-forum-post, blog, reports, articles, changelogs, source-code, release notes etc.

Maybe we can define a dictionary of types, and add a generic WEB type. Similar to how the OSV schema does it: https://ossf.github.io/osv-schema/#references-field

supportedReleases

There's two reasons I want this field:

  1. is to make it easy for publishers to publish anything over nothing.
  2. Make it easier for scanners to check for supported versions.

jQuery could, for eg, just publish this today:

{
	"products": [{
		"name": "jQuery",
		"supportedReleases": [
			"3.6.3"
		]
	}]
}

You could even use the NPM yank status field, to generate this easily for other projects (say react or electron). Any downstream scanner can rely on this field and implement it easily.

Without this, it becomes much harder to publish this information, because now the publisher has to worry and think about release cycles, and dates.

{
	"products": [{
		"name": "jQuery",
		"supportedReleases": [
			"3.6.3"
		]
		"releaseCycles": [{
			"id": "3",
			"eol": false,
			"latestVersion": "3.6.3"
		},{
			"id": "2",
			"eol": true,
			"latestVersion": "2.2.4"
		},{
			"id": "1",
			"eol": true,
			"latestVersion": "1.12.4"
		}]
	}]
}

Publishing the above requires: breaking down your releases into release cycles, deciding on EOL status for each, and finding the latest version in each cycle. It's much more work for a vendor, compared to the first. As an example, when I'd proposed publishing a table of supported electron versions with the latest version in each release cycle on the electron.org website, it was deemed as too much work (due to the regular maintainence burden and manual updates).

The big problem with the field is the ambigous "support" in "supportedVersions" - it's unclear what it means. Need to find a way to fix this:

  1. Either clarify what support means here. Such as by securitySupportedVersions etc.
  2. Provide some labelling as to what this means exactly (which subset of "support terms")

latestVersion vs latestSupportedVersions

I think the latter is more clear, as the latest information might also be published for unsupported release cycles.

@captn3m0
Copy link
Member Author

Will do a second draft considering the suggestions from here, and update soon.

// Other alternative idenfiers for this release cycle.
// Helpful especially, when the product is distributed via alternative channels
// and different identifiers are used across channels (such as platforms, countries etc).
variants: ["05.2"],
Copy link
Sponsor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have an example you can share here around variants usage?

@sa-tasche
Copy link

@captn3m0 @marcwrobel @noqcks
Hi. I would like to know if there is a schedule for bringing these changes together
and if there are any further discussions in general.
In other words: Is this going to continue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants