Approach for applying npm security patches to Docker Images

[lewisdaly]

Goal:

As a hub operator, I want regular security patches applied to my running system, so I can ensure a secure and reliable environment.

• i.e. how can we make sure docker images are upgraded?
• what are the docker best practices? Should we rebuild an image with the same tag, or add a suffix?
  • e.g. 10.1.0 or 10.1.0-patch1
  • maybe a patch is just for dependencies (we need to make rules about this)
  • other code changes can be a BUGFIX upgrade
• what do we want by default? Security patches or surety of code?

Tasks:

• Propose a model that makes prompts to Hub Operators when a new image with only patches applied is available

Acceptance Criteria:

• A proposal for a service that helps prompt Hub operators of updated patched images is made.

Pull Requests:

• TBD

Follow-up:

• N/A

Dependencies:

• N/A

Accountability:

• Owner: TBC
• QA/Review: TBC

[godfreykutumela]

I suggest we approach this from two perspective - 1. Capacitywise we may have to make a decision to support the last 3 or 4 versions with security patches but still support on more versions functionality wise. 2 Where fixing security vulnerability is only through version upgrade for a specific version then we will have to explore how that vulnerability can be mitigated. @elnyry

[lewisdaly]

I think there's 2 considerations here, with respect to the developer experience:

1. when running helm deploy, which docker images are installed by default? Are they the images that were released at the same time as the deployment? (which could be weeks or months ago) Or have they been patched in the meantime (without a new helm deployment)?

2. Is there a way for an administrator to run a command to apply the latest security patches?

helm upgrade would only work if a new helm release is made, and I don't think will update images in place if the tag hasn't changed (even if the image digest has). What if a user doesn't want to upgrade to the latest deployment, but wants to apply the security patches?

[lewisdaly]

@elnyry here's a thread I found on re-pulling updated images for security updates: kubernetes/kubernetes#33664

[godfreykutumela]

Noted @lewisdaly expanding your second point a bit, I think we should explore how we can build a way for an administrator to run a command to first check for vulnerabilities on a particular realise\version and also have an option to apply the latest security patches where they exist . From a risk perspective, we should explore a way where we can put a risk score or ratings per version for the adopters make their decision accordinly to the risk level they're willing to accept.

[elnyry-sam-k]

Thanks @lewisdaly , @godfreykutumela for the inputs and thoughts, I'll go through these resources and your inputs and suggest a guidance here - will continue with the versioning team..

[elnyry-sam-k]

Discussed with Lewis, Matt and Miguel to use an Operator pattern. @matdehaast also prosed using a controller pattern which may be simpler, investigating this.

[elnyry-sam-k]

Some notes (from Lewis's mid-pi review presentation)

1. Investigating best practices around rebuilding docker images with security patches, and how to apply them in running Kubernetes clusters
2. Images to be with a new security patch suffix: start from mojaloop/ml-api-adapter:v10.4.1; publish a new version: mojaloop/ml-api-adapter:v10.4.1.1-patch when we apply a security patch (change of node_modules only)
3. Develop a new service that runs inside a deployment; looks for updates to v10.4.1 (i.e. images with tag v10.4.1.X-patch) and then notifies Hub Operators (emails / slack notifications?) or in some cases automatically (or manually) replaces pods with patched images

[godfreykutumela]

Thanks @elnyry-sam-k and I will chat to @lewisdaly regarding follow through stories to action the 3 points from the notes above.

[elnyry-sam-k]

Thanks @godfreykutumela , I'll create a story to actually do a PoC for this service in #3 , will keep you posted.. Thanks @lewisdaly and @matdehaast

[godfreykutumela]

Okay that's perfect, thanks @elnyry-sam-k