Tracker: Adoptium Secure Software Development Framework (SSDF)

[jiekang]

Security is of critical importance to Adoptium in order to maintain the trust of its community in the integrity of the work that it does. There are evolving market requirements for software providers to meet for using secure development practices. The Adoptium SSDF, based directly on the NIST SSDF [1], represents a commitment to meet these requirements.

[1]
https://csrc.nist.gov/Projects/ssdf
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218.pdf

This issue tracks the work to be done against the SSDF framework. The intention is for individual points of the framework to be investigated, documented, and completed, to then arrive at an overall Adoptium SSDF that can be maintained into the future. I intend to create an epic issue for each point that will serve as a tracker for relevant issues across the Adoptium organization’s repositories.

Epics are split into areas following the table in [1] (page 14 onwards). A subset of entries in [1] are proposed for work here, based on relevance to the Adoptium project. Discussion and debate is open for the entire process, as well as the reference framework Adoptium chooses to follow.

Other resources of interest include:
CNCF Paper on Secure Supply Chain:
https://github.com/cncf/tag-security/blob/main/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf
Supply Chain Levels for Software Artifacts:
https://slsa.dev/

I believe the underlying work will be similar regardless of the framework it is organized under, and am proposing to follow the NIST SSDF, at least to drive the initial work towards secure Adoptium releases. The work is open for any willing contributors, and Red Hat contributors will participate.

Epics:

PO: Prepare the organization (#122)

• PO.1.1: Identify and document all security requirements for Adoptium's software development infrastructures and processes, and maintain the requirements over time.
• PO.2.1: Create new roles and alter responsibilities for existing roles as needed to encompass all parts of the SSDF. Periodically review and maintain the defined roles and responsibilities, updating them as needed.
• PO.2.3: Obtain leadership commitment to secure development and convey that commitment to all with development related roles and responsibilities
• PO.3.1 Specify which tools or tool types must or should be included in each toolchain to mitigate identified risks, as well as how the toolchain components are to be integrated with each other
• PO.3.2 Follow recommended security practices to deploy, operate, and maintain tools and toolchains
• PO.4.1 Define criteria for software security checks

PS: Protect software (#123)

• PS.1.1: Store all forms of code, including source code, executable code, and configuration-as-code, based on the principle of least privilege so that only authorized personnel, tools, services, etc. have access
• PS.2.1: Make software integrity verification information available to all users
• PS.3.1: Securely archive the necessary files and supporting data (e.g., integrity verification information, provenance data) to be retained for each software release.
• PS.3.2: Collect, safeguard, maintain, and share provenance data for all components of each software release via a software bill of materials [SBOM]

PW: Produce well secured software (#124)

• PW.1.1: Use forms of risk modeling, such as threat modeling, attack modeling, or attack surface mapping, to help assess the security risks for Adoptium
• PW.1.3: Use standardized security features and services instead of creating proprietary implementations
• PW.2.1: Review the security architecture
• PW.4.1: Acquire and maintain well-secured software components
• PW.4.2: Create and maintain well-secured software components
• PW.5.1: Follow secure coding practices that are appropriate to the development languages and environment
• PW.6.1: Use securely configured compilers and build tools
• PW.7.1: Conduct peer code reviews
• PW.8.1: Determine executable code testing needs
• PW.8.2: Scope the testing, design the tests, perform the testing, and document the results

RV: Assess, prioritize and remediate vulnerabilities (#125)

• RV.1.1: Gather information on potential vulnerabilities
• RV.1.3: Have a team and process in place to manage vulnerability reports and incidents
• RV.2.1: Analyze each vulnerability
• RV.2.2: Plan and implement risk responses for vulnerabilities
• RV.3.1: Conduct root cause analysis
• RV.3.2: Mitigate root causes
• RV.3.3: Proactively check for similar vulnerability instances

As well as the areas above that directly follow the NIST SSDF, I propose an area for additional items for Adoptium

A: Secure Adoptium (#126)

• A.1.1: Defined support & lifecycle maintenance plan
• A.1.2: Monitor and ensure the integrity of customer-facing repositories
• A.2.1: Identify layered dependencies
• A.2.2: Track Provenance and Pedigree
• A.3.1: Data lifecycle management
• A.3.2: Data lifecycle validation
• A.4.1: Conduct code scanning

[tellison]

Thank you for kicking off this initiative @jiekang. It has already been discussed at the PMC, where there was enthusiastic support for this work.

I see you have already got some good level of detail with the approach and areas for Adoptium to address. I think it would be helpful to spend a short time introducing this topic to the community and ensuring an understanding for the direction proposed. As you wrote above:

Discussion and debate is open for the entire process, as well as the reference framework Adoptium chooses to follow.

Would you like to introduce this in the #general channel as a starting point? Once underway it may well warrant a specific channel for SSDF discussions, but let's start there.

[smlambert]

Adding this comment, based on discussions in the March 10, 2022 Steering Committee call, and as a reminder and invitation for members to provide input and feedback.

Most of our expected activities will be documenting what we already do at the project, using the NIST SSDF outline as a guide. We have selected the US government based NIST framework as a model to follow in this exercise as it seems thorough and rigorous. We recognize there may be other models and are open to input if there are other references that we should consider as part of this work (especially if they are more rigorous, though an initial scan indicated that the NIST SSDF to be satisfyingly thorough, we did look at BSA and SLSA v0.1).

The checklist provided by the SSDF framework encompasses many activities that we already do, or have initiated at the project, so much of the effort related to this issue will take the form of assessment, documentation and identifying whether there are any gaps that the technical teams should try to fill. We believe this effort will help to further stabilize the project and is a step along our path of continuously improving how we build, test, and deliver software. Hopefully, as we identify and clearly articulate areas of improvement, we can engage experts from our community to join this effort.

As a personal observation, I also believe that this effort is not just of value to Adoptium, but to the ecosystem at large and I will be happy to see and share the progress more broadly.

EDIT SL/Nov23: To clarify, we are using SSDF and SLSA as frameworks that guide us in our mission to ensure secure software development. In most cases, the criteria of each framework overlaps. If there are unique criteria in one or the other framework, we will attempt to achieve them, but first focussing on the overlap criteria present in both.

[sxa]

Important documentation for initial compliance - this information should ultimately be put under version control in this repository.

• SSDF PS 1.1 - List of customer facing repositories - [SSDF] [PS1.1] - Document Project Repositories & Access #143
• SSDF PS 1.2 - temurin ACLs to be reviewed annually - more security detail in https://docs.google.com/document/d/1w3znf2X4y0yoiK2w1cNxSwu8ok7ibWCbD3t1oEs2wwU/edit#
• Will need to sanitise and document Jenkins ACLs Audit all ACLs on the jenkins server infrastructure#2814
• SSDF RV - We are following the eclipse process as per https://www.eclipse.org/security (detailed process info) and Adoptium has a representative on the security team. This repository has the security policy documented too.

[sxa]

For reference, we have recently published two blog posts on the secure development processes:

• SLSA level 2 compliance for Eclipse Temurin (We have a badge on the project page now)
• SSDF activities at Adoptium

[sxa]

Plan of attack for some of the more important parts of SSDF during 2023. Note that I've tentatively planned in two months for each phase of this, so the estimated target dates against each plan reflect that:

Done

• Most of RV - based on Eclipse Vulnerability reporting
• PS2.1 - Software integrity information - we have hashes and signatures for product downloads
• PS1.1 - Repositories and access control documented: [SSDF] [PS1.1] - Document Project Repositories & Access #143
• PO2 - Leadership commitment to secure dev. Agreement to progress was given by PMC last year (Slide 13, 2023 bullet point 1 of the approved program plan)
• PW7.1/7.2 - Code review policy (GitHub reviewers - 2 on critical repos implemented. Find reference)

Phase 1 (2023-05)

• PS3.2 (bld) Improve SBOM component lists using (probably) strace on Linux and other tools elsewhere. Dependencies for Windows Linux Mac (Look at others later? Base on Linux work?) Also Populate SBOM with minimal properties to enable reproducible-build temurin-build#3314 (AL+friends)
• PW6.1(a)(infra+bld) Securely configure build tools. Define policy for patching compilers etc. (Consider reproducibility - implement in phase 2) Define policy for compiler upgrades on our machines infrastructure#2966
• PW5.1 (infra+bld+test) Linter config and update policy. Input validation on processes needed? Define/document linter policy and other validation checks on automation infrastructure#2971
• PW1.2 (part1) (infra) Define patching policy for servers and critical infrastructure e.g. jenkins. Identify threat model = attack vectors Secure the public Jenkins server infrastructure#2108 (Use jenkins as model for others - define regular patching cycle) - SXA/MV to handle jenkins Also Define and implement patching policy for critical infrastructure infrastructure#2969
• PO5.1 (infra) - Track all operating systems/distribution versions to ensure we can keep them secure: Collate data on OS+distribution support dates so we can make infrastructure decisions infrastructure#2902
• PO5.1(a) (infra+bld) - Network segregation for all Linux systems (s390x to go: Support and use docker builds on Linux/s390x infrastructure#2731) - SXA
• PO5.1 (infra+bld) Remove/document+checksum any downloads other than from github.com/adoptium during build Investigate how to make build more secure when download/install tools during build temurin-build#3096, Add GPG signature verification to adoptium binary downloads infrastructure#2908 HK
• PO5.1 (infra) Identify tooling for and define access logging policies. Identify tooling for auditing access controls to infrastructure infrastructure#2968 SF
• PO5.2 (infra) - Document policy for granting/revoking access to team members (systems/infra/providers/bastillion/nagios) and temporary access. Ensure all policies for access control additions/removal are documented infrastructure#2967 - SXA
• PO3.2 (infra) Define and implement process for deploying toolchains. Ansible auto-deployment HK
• BitWarden migration from LastPass Migrate infrastructure password management to BitWarden infrastructure#2951 SXA

Phase 2

• PW1.2 (Part2) (infra) Implement patching policy for servers and critical infrastructure . Follows on from 2108 in phase 1: Define and implement patching policy for critical infrastructure infrastructure#2969 SXA
• PW2.1 (Infra) Security architecture review - investigation has been done under [SSDF Issue] PW2.1: Review the security architecture #144
• PO1 (infra) Ensure no gaps in security policy documentation SXA
• PW6.1(b) (infra+bld) Policy for critical component (e.g. compiler) updates for security and reproducibility - follow-on to Define policy for compiler upgrades on our machines infrastructure#2966. SSDF PW6.1: Documentation of build servers, compilers, tools, build process infrastructure#2503 (Also requires identifying the list of components which will come from the phase 1 SBOM updates)
• PW1.3 - Define and document policies and ACLs for identity and security management (Mostly our somewhat neglected Bastillion) Fully document Bastillion use/rollout/roles etc. infrastructure#2970 - documented in doc: Add document describing access controls at the project infrastructure#3002
• PW5.1/5.2 - Determine if more testing needs to be done on the build/infra tooling: SSDF PW5.2: Documentation of testing for infra tools infrastructure#2502 / See also
• PO5.1 Change control tooling/process for direct machine accesses. Logging under Mechanism for tracing direct machine logins infrastructure#3212
• PO5.1 - Feasibility of network segregation for build systems for non-Linux systems: Windows/Mac/Solaris/AIX (Docker/VM/zone/WPAR?) Investigate feasibility of ephemeral build systems for non-Linux platforms temurin-build#3264.
• PO5.1 Since we're using Linux images via DockerHub, can we do verifiable checksums on our images when they are pushed up / retrieved? Can we create verifiable checksums for our dockerBuild images? infrastructure#2734
• PO5.1 (bld/infra) Define and implement policy to monitor all prereq software identified in phase 1 for new vulnerabilities and have a suitable patching policy Define policy for identifying security issues in prerequisite software products temurin-build#3265 - REQUIRES COMPLETE SBOM
• PW1.1 - Conduct threat model analysis with a view to defining actions for phase 3 (Follow-on to Identify tooling for auditing access controls to infrastructure infrastructure#2968): EPIC: Define & Implement Access Auditing Policy & Tools infrastructure#3047 SF
• PO5.2 (infra) Deployment of MFA where possible for all parts of infrastructure (including ssh keys) and document exceptions MFA implemented for infra providers - exceptions to be documented SXA
• policy defined for DockerStatic deployments - HK

Phase 3

• PO5.2 (bld) Determine feasibility of isolating development and production build systems.
• PO5.2 (infra+test) Principle of least privilege. To include investigation of Grinder jobs' and access/threat modelling
• PO5.1 (Infra) Ensure no host-based authentication. Define process for distributing ssh host keys
• PS3.1 (infra+bld) Secure downloads (Should be RO for some, zero access for most) (See [SSDF Epic] PS: Protect software #123 (comment))
• PS3.2 (infra+bld) Full SBOM with list of all dependencies for Temurin on all platforms Ref: SSDF PO.3.2 Documentation on maintainance and security of toolchains infrastructure#2553 (See also [SSDF Epic] PS: Protect software #123 (comment))
• PW 1.2 (infra) Threat model mitigation based on the outcomes of analysis in earlier phase
• PO4.1 - Define criteria for security checks (verify downloads and signatures once published? Verify reproducibiility?)
• Finalise what to do with secrets repository and associated tooling) Evaluate dotgpg alternatives for adoptopenjdk/secrets infrastructure#2661 - Some credentials already migrated, rest can be considered here.

Phase 4

• PS4.1 (bld) Provenance (SBOM) for all dependencies. Store blessed versions in local repo for availability?
• PW8 - Penetration testing
• PW9 - Secure, non-exploitable options by default.
• PO4.2 - Implement security checks defined in PO4.1 in previous phase
• PO5.2 - Isolate development and production build systems based on analysis in previous phase

[tellison]

Plan of attack for some of the more important parts of SSDF:

Done

• Most of RV - based on Eclipse Vulnerability reporting
• PS2.1 - Software integrity information - we have hashes and signatures for product downloads
• PS1.1 - Repositories and access control documented: [SSDF] [PS1.1] - Document Project Repositories & Access #143
• PO2 - Leadership commitment to secure dev. Agreement to progress was given by PMC last year (Link?)

Maybe point to the approved project plan? I've just uploaded the latest version via the WG mailing list.

• PW7.1/7.2 - Code review policy (GitHub reviewers - 2 on critical repos implemented. Find reference)

Will be helpful when the EF declarative policies and permissions code is rolled out for repos. In the meantime perhaps we can list our critical repos and point to the OSSF scorecard reports.

Phase 1

• PS3.2 (bld) Improve SBOM component lists using (probably) strace on Linux and other tools elsewhere. Dependencies for Windows Linux Mac (Look at others later? Base on Linux work?)
• PO5.1 (infra) - Track all operating systems/distribution versions to ensure we can keep them secure: Collate data on OS+distribution support dates so we can make infrastructure decisions infrastructure#2902
• PW6.1(infra+bld) Securely configure build tools. Define policy for patching compilers etc. (Consider reproducibility - implement in phase 2) Define policy for compiler upgrades on our machines infrastructure#2966
• PW5.1 (infra+bld+test) Linter config and update policy. Input validation on processes needed? Define/document linter policy and other validation checks on automation infrastructure#2971
• PW1.2 (part1) (infra) Define patching policy for servers and critical infrastructure e.g. jenkins. Identify threat model = attack vectors Secure the public Jenkins server infrastructure#2108 (Use jenkins as model for others - define regular patching cycle)
• PO5.1(a) (infra+bld) - Network segregation for all Linux systems (s390x to go: Support and use docker builds on Linux/s390x infrastructure#2731).
• PO5.1 (infra+bld) Remove/document+checksum any downloads other than from github.com/adoptium during build Investigate how to make build more secure when download/install tools during build temurin-build#3096
• PO5.2 (infra) - Document policy for granting/revoking access to team members (systems/infra/providers/bastillion/nagios) and temporary access. Ensure all policies for access control additions/removal are documented infrastructure#2967
• PO3.2 (infra) Define and implement process for deploying toolchains. Ansible auto-deployment and policy defined for DockerStatic deployments
• PO5.1 (infra) Identify tooling for and define access logging policies. Identify tooling for auditing access controls to infrastructure infrastructure#2968
• (Also finalise what to do with secrets repository and associated tooling) Migrate secrets repository from adoptopenjdk to adoptium infrastructure#2661

+1 to all those.

Phase 2

• PW1.2 (Part2) (infra) Implement patching policy for servers and critical infrastructure . Follows on from 2108 in phase 1: Define and implement patching policy for critical infrastructure infrastructure#2969
• PW2.1 (nfra) Security architecture review - investigation has been done under [SSDF Issue] PW2.1: Review the security architecture #144
• PO1 (infra) Ensure no gaps in security policy documentation
• PW6.1 (infra+bld) Policy for critical component (e.g. compiler) updates for security and reproducibility - follow-on to Define policy for compiler upgrades on our machines infrastructure#2966. SSDF PW6.1: Documentation of build servers, compilers, tools, build process infrastructure#2503 (Also requires identifying the list of components)
• PO5.1 Change control tooling/process for direct machine accesses.
• PW1.3 - Define and document policies and ACLs for identity and security management (Mostly our somewhat neglected Bastillion) Fully document Bastillion use/rollout/roles etc. infrastructure#2970
• PW5.1 - Determine if more testing needs to be done on the build/infra tooling: SSDF PW5.2: Documentation of testing for infra tools infrastructure#2502 / See also PW5.2 - secure coding practices
• PO5.1 - Feasibility of network segregation for build systems for non-Linux systems: Windows/Mac/Solaris/AIX (Docker/VM/zone/WPAR?) Investigate feasibility of ephemeral build systems for non-Linux platforms temurin-build#3264.
• PO5.1 Since we're using Linux images via DockerHub, can we do verifiable checksums on our images when they are pushed up / retrieved? Can we create verifiable checksums for our dockerBuild images? infrastructure#2734
• PO5.1 (bld/infra) Define and implement policy to monitor all prereq software identified in phase 1 for new vulnerabilities and have a suitable patching policy Define policy for identifying security issues in prerequisite software products temurin-build#3265
• PW1.1 - Conduct threat model analysis with a view to defining actions for phase 3 (Follow-on to Identify tooling for auditing access controls to infrastructure infrastructure#2968)

+1 again.

Phase 3

• PO5.2 (infra) Deployment of MFA where possible for all parts of infrastructure (including ssh keys) and document exceptions

Consider moving to earlier phase?

• PO5.2 (bld) Determine feasibility of isolating development and production build systems.
• PO5.2 (infra+test) Principle of least privilege. To include investigation of Grinder jobs' and access/threat modelling
• PO5.1 (Infra) Ensure no host-based authentication. Define process for distributing ssh host keys
• PS3.1 (infra+bld) Secure downloads (Should be RO for some, zero access for most) (See [SSDF Epic] PS: Protect software #123 (comment))
• PS3.2 (infra+bld) Full SBOM with list of all dependencies for Temurin on all platforms Ref: SSDF PO.3.2 Documentation on maintainance and security of toolchains infrastructure#2553 (See also [SSDF Epic] PS: Protect software #123 (comment))
• PW 1.2 (infra) Thread model mitigation based on the outcomes of analysis in earlier phase
• PO4.1 - Define criteria for security checks (verify downloads and signatures once published? Verify reproducibiility?)

Phase 4

• PS4.1 (bld) Provenance (SBOM) for all dependencies. Store blessed versions in local repo for availability?
• PW8 - Penetration testing
• PW9 - Secure, non-exploitable options by default.
• PO4.2 - Implement security checks defined in PO4.1 in previous phase
• PO5.2 - Isolate development and production build systems based on analysis in previous phase

Happy with those. Lots of work!

[sxa]

@tellison The MFA one for infra was mostly in phase 3 because there was quite a lot in 2 and I didn't want to take anything out, but I'm shifting it up to phase 2 so that it is at least started earlier. This will mean it can be more swiftly included in the documentation of onboarding/offboarding which is part of Phase 1.

Added link to the plan now that it's available :-)

On the OSSF scorecards is there a specific artefact produced from those or are you'd like to point at or would it just be the executions of the actions workflows?