feature: on demand docker environments

[saikonen]

Adds support for on demand Docker images. Open for discussion

open todo's

• add fallback to conda environment when executing locally with --environment docker
• add support for imagebakery endpoint auth
• cache bakery image tags locally to avoid environment drift with consecutive deployments when using loosely pinned versions

[savingoyal]

python for.py --environment=docker run without @kubernetes immediately returns an error that Image Bakery is not configured. Perhaps it shouldn't throw the error?

[savingoyal]

the imagebakery-cache can also be made an optimization detail within docker_environment.py

[savingoyal]

also, can you help me understand the contents of this cache?

[saikonen]

moved caching to the environment side. what is in the cache at the moment is

• a unique hash derived from the requested environment (python, packages, image-type etc.)
• the image tag that was returned from the bakery endpoint for the request
• we also persist the request payload that was sent to the bakery in order to be able to list details of cached images via docker cache list

[savingoyal]

Would something of this sorts work here -

Suggested change

	import json
	import requests
	class FastBakery(object):
	def __init__(self, url):
	self.url = url
	def bake(
	self,
	python_ver=None, # TODO - Ensure the keys are same in the API spec and client
	pypi_packages=None,
	conda_packages=None,
	base_image=None,
	image_kind="esgz-zstd",
	):
	deep_clean = (lambda f: lambda d: f(f, d))(
	lambda self, item: {
	k: v
	for k, v in (
	(k, self(self, v)) if isinstance(v, dict) else (k, v)
	for k, v in item.items()
	)
	if v is not None and v != {}
	}
	)
	payload = deep_clean(
	{
	"pythonVersion": python_ver,
	"imageKind": image_kind,
	"pipRequirements": pypi_packages,
	"condaMatchspecs": conda_packages,
	"baseImage": {"imageReference": base_image},
	}
	)
	return self.aws_iam_invoker(payload)
	def aws_iam_invoker(self, payload):
	try:
	# AWS_IAM requires a signed request to be made
	headers = {"Content-Type": "application/json"}
	payload = json.dumps(payload)
	import boto3
	from botocore.auth import SigV4Auth
	from botocore.awsrequest import AWSRequest
	from botocore.exceptions import BotoCoreError, NoCredentialsError
	import urllib
	session = boto3.Session()
	credentials = session.get_credentials().get_frozen_credentials()
	# credits to https://github.com/boto/botocore/issues/1784#issuecomment-659132830,
	# We need to jump through some hoops when calling the endpoint with IAM auth
	# as botocore does not offer a direct utility for signing arbitrary requests
	req = AWSRequest("POST", self.url, headers, payload)
	SigV4Auth(
	credentials, service_name="lambda", region_name=session.region_name
	).add_auth(req)
	response = requests.post(req.url, data=req.body, headers=req.headers)
	# Manually check the status code and raise an exception if it's an error
	if response.status_code >= 400:
	raise requests.exceptions.HTTPError(
	f"HTTP {response.status_code} Error: {response.reason}\nResponse Body: {response.text}",
	response=response
	)
	return response
	except requests.exceptions.HTTPError as e:
	raise e
	except (requests.exceptions.RequestException, BotoCoreError, NoCredentialsError) as e:
	# Handle specific errors related to requests and AWS services
	raise Exception(f"Failed to invoke AWS service due to network or AWS error: {e}")
	except Exception as e:
	# Handle other possible exceptions
	raise Exception(f"An unexpected error occurred: {e}")

[savingoyal]

these comments too

[savingoyal]

multiple steps might share the same image - maybe we can check if that is the case and only bake unique images?

[saikonen]

the images are baked in sequence so the consecutive steps with identical packages should be hitting the cache at this point, skipping the baking. previously the caching was happening inside the fast_bakery, but moved to the environment side now so it should be more clear.

[savingoyal]

we can avoid introducing these as env vars here and in @kubernetes with the expectation that using fast bakery may involve using @secrets or @environment - to configure the URL and auth. The client can try to read from env vars unless explicitly configured with the URL and auth?