Decide on a deprecation strategy for migrating the code to a new Go-based TUF implementation - rdimitrov/go-tuf-metadata

[rdimitrov]

The following issue is a placeholder for discussing the migration strategy of deprecating the current go-tuf code base in favor of https://github.com/rdimitrov/go-tuf-metadata.

Reason
The reasoning behind this is the discussions we've been having in the go-tuf community for the past 1+ years regarding the overall maintainability and usability of the current implementation. There are several less-than-optimal design decisions in go-tuf which often cause a lot of unnecessary implications both for the maintainers of the project but also for the actual users. Adding a feature or a simple bug fix (i.e. #272) sometimes ends up being surprisingly difficult to implement and review, and unfortunately, that makes it much harder to attract new contributions too.

Python-tuf was in the same situation and its community decided to do the same thing successfully redesigning its codebase approx. 2 years ago. The go-tuf community got inspired by this and evaluated that it would be worth to try and apply the same design principles used in python-tuf to create a new Go-based TUF implementation.

The result of this effort is the https://github.com/rdimitrov/go-tuf-metadata project.

Another example of this approach was developed in parallel - https://github.com/theupdateframework/tuf-js, which is a javascript-based implementation of TUF again based on the python-tuf design.

Goal
The end goal is to deprecate the current go-tuf codebase and replace it with the one from https://github.com/rdimitrov/go-tuf-metadata, so we continue to use the same repository location. The following are some of the things we need to complete in not order to accomplish that:

• Decide on how to begin announcing this intention. Maybe have N releases (2 should be enough but maybe more?) where we say that users should expect this change in the future? Perhaps, once we have a rough estimation we can put a date on which the new release will happen?
• For how long should we support the old version after the migration is completed? Should it be in a separate branch?
• The go-tuf project has several things in it. On one side there's the go-tuf package, then there are client and repository-side CLI tools but also other miscellaneous things. We should assess what needs to be abstracted/kept away from the current code base (like Move the encrypted package away from go-tuf #476)
• rdimitrov/go-tuf-metadata should have proper test coverage and documentation
• Should we provide, and thus implement the CLI tools too using the new metadata implementation? If so, where should their code reside? Same or a separate repository?
• Prepare proof-of-concept implementations for some existing clients of go-tuf using https://github.com/rdimitrov/go-tuf-metadata. This should help justify and verify such API transition. There are already a few things like that for sigstore's cosign and sigstore library.

These are just some of the action items, but of course, more will be added in the process of this discussion.

[udf2457]

As long as we finally get more flexible key support, including PKCS#11 (#427) and other options (#525), maybe even Yubikeys then you've got my +1 @rdimitrov 😉

Also strikes me as sensible to aim to kill the dependency on secure-systems-lab/securesystemslib in favour of go stdlib crypto.

[MDr164]

As long as we finally get more flexible key support, including PKCS#11 (#427) and other options (#525), maybe even Yubikeys then you've got my +1 @rdimitrov 😉

Also strikes me as sensible to aim to kill the dependency on secure-systems-lab/securesystemslib in favour of go stdlib crypto.

As far as I know the new code base only uses the securesystemslib for CJSON which are just 150 line of rather simple Go. It also seems like the stdlib JSON parser as no validator for CJSON either. I'm fine with keeping this rather small dependency as this lib really just uses stdlib for under the hood for CJSON and not copy over the go into go-tuf.

For Yubikeys it should just be a matter of talking to the Yubikey PIV and use PKCS11, could also be done by using an outside tool to get the crypto token and feed it into stdin maybe?

[udf2457]

@MDr164 Thanks for the reply.

If securesystemslib is only used for CJSON, then why was I referred back to securesystemslib when bringing up the lack of PKCS11 support before in #427 ? See comment from @trishankatdatadog.

[MDr164]

The current/older code base also uses the lib for the encrypted package but that might even be moved to a different repository as outline in the issue text and in #476.

EDIT: And for the issue you linked I suppose the idea is to have specialized implementations of crypto algorithms not inside the go-tuf repository but rather in the securesystemslib though I would also prefer to use stdlib crypto or golang.org/x/crypto in case the sought after implementation already exists to a usable degree. If not, pulling in securesystemslib would still be the more viable solution, no?

[MDr164]

So for the migration I propose following steps:

1. Get test coverage up on new repo and let it undergo some reviews
2. Push old go-tuf code into a release branch
   2.1. Due to go-tuf not having a major release yet we should create a minor release branch, e.g. v0.6 as of right now
   2.2 Continue fixes as needed on legacy branch and bump patch version for releases as needed
3. Merge new code base to main branch and create now minor release, e.g. v0.7.0 as of right now

This way we would be able to support both the legacy code and the new code base and users would just have to depend on the respective minor release. Due to not being bound to API guarantees with a v0 major version this would be allowed according to semantic version conventions. Though we should announce this change as to not surprise anyone.

[trishankatdatadog]

Agree: let's do this

[rdimitrov]

So for the migration I propose following steps:

1. Get test coverage up on new repo and let it undergo some reviews
2. Push old go-tuf code into a release branch
   2.1. Due to go-tuf not having a major release yet we should create a minor release branch, e.g. v0.6 as of right now
   2.2 Continue fixes as needed on legacy branch and bump patch version for releases as needed
3. Merge new code base to main branch and create now minor release, e.g. v0.7.0 as of right now

This way we would be able to support both the legacy code and the new code base and users would just have to depend on the respective minor release. Due to not being bound to API guarantees with a v0 major version this would be allowed according to semantic version conventions. Though we should announce this change as to not surprise anyone.

Thanks for this @MDr164 💯

At today's go-tuf meeting we agreed to cut a release having the latest go-tuf state as of now (v0.7.0) next Monday and create a branch with it after that.

After that we'll be free to proceed with step 3. We can even sync about it if you'd like to help with that 👍

[MDr164]

Sounds good, I'll be at tomorrows TUF general sync as well. In addition I'm currently preparing to cut a release for my software that consumes go-tuf as well so this aligns well. Also got some WIP code for the unit tests I still need to put up so I know what deadline I gotta work with 👍

[rdimitrov]

Sounds good, I'll be at tomorrows TUF general sync as well. In addition I'm currently preparing to cut a release for my software that consumes go-tuf as well so this aligns well. Also got some WIP code for the unit tests I still need to put up so I know what deadline I gotta work with 👍

Awesome! It's not a hard deadline though so we can adapt if needed 👍

[HLeithner]

@rdimitrov would you like to pin this issue to the repo so people coming here can see the motivation and the way to go at the first glance?

[rdimitrov]

@rdimitrov would you like to pin this issue to the repo so people coming here can see the motivation and the way to go at the first glance?

Thanks! Created a PR about it - #569

[kommendorkapten]

I was talking with @codysoyland yesterday about this. He proposed creating a new go-tuf-medadata version as "v2" by using Go semantic tags? I.e using module github.com/theupdateframework/go-tuf/v2 in the go.mod. This way we wont break any clients, then we can keep all code up to v0.7.0 in that branch, and performes "v1" releases from that. With this structure, we should be able to replace entire content of the master branch.

Maybe this has been proposed earlier and I missed that.

[mnm678]

@kommendorkapten is going to explore the "v2" tag option and create steps to perform this release

[rdimitrov]

Reference of updating the CI to run for the branched version - sigstore/cosign@43bde0e

[rdimitrov]

I'll keep this one opened until we:

• updated the CI so dependabot continues to push updates to the legacy code base
• deprecate the rdimitrov/go-tuf-metadata repo