Feat: rework Container image, Compose manifest and CI #64

almereyda · 2024-05-09T20:27:49Z

This change request presents a major opinionated rewrite of this repository.

adds the list of authors
adds a LICENSE
adds a dot env file .env to configure Pretalx as a container-native 12 factor app
adds cron to the image to be ran in a separate container independently from the host system
adds a separate Nginx container to terminate static/ assets and media/
adds an example to run Pretalx with plugins
adds different Compose manifests for different scenarios, including legacy Docker Compose
adds a generic build workflow to easily build different images
adds a plugins workflow to build a pretalx-extended image

updates CI
updates the README
updates the database to recommended PostgreSQL
updates the default image name to pretalx/pretalx in CI and Compose
updates the Pretalx container image from a source build to a release build
updates the entrypoint to for regular cases, removes automigrate/-rebuild
updates the now unused pretalx.cfg example to be compatible with the changed setup

removes supervisor from the container and replaces its function with individual containers per job
removes unmaintained reverse proxy examples
removes unrelated Ansible
removes linux/arm64 build target

closes #21
closes #56
closes #59 ?
closes #62

almereyda · 2024-05-09T20:28:00Z

This PR presents the result of 16 hrs of work to reshape it into a more "production-ready" form. This is achieved by dropping the support for local source builds.

Local container builds for developing Pretalx should be implemented with development containers in the upstream repository.

Additionally, following the 12factor.net approach of designing web applications, it is useful to consider the parts of the application self-contained and independent from external systems. Why the reliance on a cron scheduler may be defeating the purpose of separating concerns in containers.

A convention designed for running jobs in distributed systems is to use a worker queue and have it be triggered with timers. Fortunately Celery is already used for dispatching asynchronous tasks. Its Celery Beat feature can also be used to regularly schedule jobs, removing the need for a separate cron container.

I'm especially invested in this, since the cron implementation took a quarter of the time from the overall procedure and is prone to fail in a container.

What could possibly go wrong?

Celery Beat is to be implemented upstream as well.

To complete the move towards "cloud-native", distributed application design, it is also advisable to implement the database URL pattern, which had been introduced by Heroku and is of good ease, convenience and use to DevOps people.

jazzband/dj-database-url is to be implemented in Pretalx.

Further on, now that the application setup is more decoupled, images are more lightweight and deterministic, due to switching to pinned release versions of Pretalx, it becomes possible to imagine to further

automate the image build pipeline, e.g. by reacting to upstream releases and to offer versioned, tagged releases here, too.

I'm thinking of the way how these two repositories react to their upstream, adapted from Node to Python:

In case you agree with the follow ups as outlined above, I'd step ahead and open respective issues.

almereyda · 2024-05-09T21:42:27Z

More inspiration could be taken from allmende/docker-fiduswriter.

context/default/Dockerfile

DASPRiD · 2024-05-09T21:56:20Z

context/default/entrypoint.sh

+export PRETALX_DATA_DIR="${PRETALX_DATA_DIR:-/data}"
+
+PRETALX_FILESYSTEM_LOGS="${PRETALX_FILESYSTEM_LOGS:-/data/logs}"
+PRETALX_FILESYSTEM_MEDIA="${PRETALX_FILESYSTEM_MEDIA:-/data/media}"


Shouldn't these defaults match what's set in the .env.example?

This was adapted from the original image and I guess for now they do not show the container defaults, but rather the application defaults.

I'm open to hearing further opinions on this.

DASPRiD · 2024-05-09T22:04:24Z

This looks generally good to me.

While in a Kubernetes environment I'm used to having CronTab jobs running periodic stuff in containers, I don't see too much harm in having crontab running within the application container. That is, unless someone wants to create replicas of the application container and run the worker things isolated on their own, but I doubt that this would be possible with Pretalx, so this is likely a non issue.

For static file serving I think that nginx as a separate container (or any other simple file server like halverneus/static-file-server) is a good choice, instead of the main container doing the serving.

👍 Overall, I'm happy with these changes.

Could we maybe consider having a GitHub workflow which automates new releases to DockerHub (or GitHub Packages)? This would avoid having bug-fixes merged into main but then not released for several months.

partially reverts a00895e

almereyda · 2024-05-13T12:06:13Z

Sorry for cd06a80, cd06a80, 68537d4 plus the additional pull requests from the allmende namespace mentioned in the merge commits:

They happened when working in multiple environments for the original Compose, the legacy Compose and the CI adaptations, which are always a bit "verbose" in commits, due to how their side-effects are only produced and can only be tested by pushing and inspecting the response of the CI system.

rixx · 2024-05-15T15:07:27Z

@almereyda Thank you for all this work and time invested in this repository! Given your interest and time investment, I have to ask – would you be interested in becoming a maintainer for this repository?

almereyda · 2024-05-16T12:11:57Z

Hi @rixx, thanks for the kind question. Actually, this was a one-shot effort, but given others like @DASPRiD would like to share the effort, I can imagine to occasionally review PRs and merge them. The current maintainers are the people listed as organisation members, am I correct?

For now I see a few obstacles on the path ahead for the breaking changes in the rework as present:

What is the migration path for people who are using the pretalx/standalone image? Should we move towards an omnibus "fat container" image with S6 as job scheduler, that integrates all jobs for testing purposes?
What is the migration path for people who are using the repository in its current state with a source build?
Will pretalx/pretalx upstream accept a PR that implements a development container together with a Compose manifest for the auxiliary containers for Postgres and Redis?

Then there are technical questions, which would be good to settle as the pretalx-docker community:

What is the conventional way to build an unprivileged Django container image for us?
There are many examples, but they all make different design decisions for setting up and optimising the build artifact.
Which kind of release and build procedure is wanted? Do we automate all the things as much as possible, including daily jobs to check for new versions upstream and automatically generating new release artifacts, if applicable? How much room for manual intervention do we leave?
Do we want to maintain the exemplary pretalx/pretalx-extended image with plugins, for the sake of having it available as an example to build upon when adding other combinations of plugins?

With the multiple layers involved, each incurs opinionated choices on how to do things and which use cases to support.

CI
Container image manifest + artifact
Compose manifest + legacy Docker Compose manifest

If the question of becoming a maintainer to be in a position to make these choices for the community without enquiry, it is not me who we are looking for. If we can keep a small overhead of a conversation about the what's, how's and why's, please count me in.

rixx · 2024-05-17T09:32:28Z

The current maintainers are the people listed as organisation members, am I correct?

No, this repository is "community maintained", which means that I chuck the permissions at everybody who wants them after contributing to the repository, as I am neither interested nor qualified to maintain anything Docker related. (The other two members of the pretalx org are largely there as bus factor alleviation, fwiw, and aren't involved in the day-to-day development of pretalx.) The current maintainers are @Lukas2112 and @shoetten.

Will pretalx/pretalx upstream accept a PR that implements a development container together with a Compose manifest for the auxiliary containers for Postgres and Redis?

Definitely not, no. I can imagine no argument in favour – as I don't know Docker, don't use Docker, and have no interest in it either, this development container would become broken/outdated almost immediately, and would increase the maintenance burden for no real gain. All Docker related things are meant to be contained in this repository.

Which kind of release and build procedure is wanted? Do we automate all the things as much as possible, including daily jobs to check for new versions upstream and automatically generating new release artifacts, if applicable? How much room for manual intervention do we leave?

I'm very much in favour of as much automation as possible. Currently, when I release a new pretalx version, I update the submodule, tag the commit, and push to kick of the build, which is fine (I have to do a bunch of manual stuff on release anyway, like releasing a blog post, so this doesn't hurt). Definitely don't want anything more manual than this – I'd be fine with an even more automated process that just checks for new pretalx releases.

If the question of becoming a maintainer to be in a position to make these choices for the community without enquiry, it is not me who we are looking for. If we can keep a small overhead of a conversation about the what's, how's and why's, please count me in.

Completely understandable – and like I said, I appreciate all the work you've done, and the maintainership offer was just that: an offer of write access to this repository. Let me know if you want it, and no hard feelings if not.

The problem is kinda that somebody has to make those decisions, and I'm not in a position to do so, either – I will say that I'm generally in favour of few breaking changes, but on the other hand, we do advertise that this repository is community maintained and not officially recommended, so there is a bit of leeway. I think at minimum, we should have an upgrade/migration guide in the README that explains what has changed, and why, and what people ought to do. Everything else (I know I skipped several of your questions), I'd leave up to you and everybody who uses this repo or is invested in it.

almereyda · 2024-05-17T12:24:55Z

Okay, I will write up a migration guide, which seems the only obstacle to introduce these breaking changes, take the privilege and we take it from there.

I will separate the question of developing from immutable artifacts in development containers into a pretalx discussion thread later down the road. There is something to gain from the immutable, container-native approach which I would like to outline in a bit more detail.

With write access to the repository, I would then attempt to further automate the build and release procedure. Renaming the repository to pretalx-oci would maybe also help to not discriminate against Podman and other container schedulers (Nomad, Kubernetes, Swarm) and allow for a single place to collect artifacts for many possible applications of the OCI images.

almereyda · 2024-05-20T21:09:12Z

Of course pretalx/pretalx@a7a8f22, raised from pretalx/pretalx#1773, now incurs the need to produce and run from a source build from main, which should be added to the chores. The current setup only expects to install pretalx from versioned releases. Are there maybe intermediary wheels from CI testing one could repurpose?

Also continuing the support for the standalone image in a versioned and source build variant remains within reach.

If there are other wishes for this reorganisation, please add them here and I will try to include them within the next cycle of activity until say end of the month.

- adds .env.build.example - adds bin/ directory with maintenance and life-cycle scripts - adds layered build stages - for the base, default, plugin-extended and cron-extended images - for standalone images - for local and remote source builds - adds explicit image contexts per build stage * updates .env.example * moves the previous image built from a submodule into context/source/standalone/Dockerfile.debian.local together with a Compose example at compose/build/source/standalone.local.yml - removes plugins build stage

…appropriate

…lled yet

…ISO date versions

almereyda · 2024-05-23T02:07:29Z

The last two days have brought new image variants, local build scripts, a build pipeline and the proposed migration hints to the README.

For now, the pipeline can be triggered manually and by releases.

The other push and PR triggers have been removed, until this has been settled a little.

The image variants reintroduce the standalone image for easing the migration in existing setups.

The extended and cron variants from the "stock" image have also been applied to the standalone one for sake of completeness. This means people can potentially remove the external dependency to cron from their existing containerised application setup.

There are also new example contexts to build the application from source, of which one mimics the current state of affairs, namely context/source/standalone/Dockerfile.debian.local, which is included in the overlay compose/build/source/standalone.local.yml.

The one interesting for me to be able to run from main, including the fix for pretalx/pretalx#1773, is compose/build/source/extended.cron.remote.yml.

The README tries to be complete and concise at the same time. Due to the multiple ways to approach the repository, its images, the Compose manifests and also the CI pipeline(s) we cannot be exhaustive here. Maybe the setup is complex enough to consider adding separate documentation to docs/.

This is now ready for merge with approaching the 50th hour. I would leave the click to another maintainer, as I wouldn't want to misuse the gained write-permission to the repository with my own and first contribution.

What's left for future cycles is roughly:

Optimisation of the build pipeline, esp. with regards to code deduplication
Automation of the build triggers, esp. with regards to upstream releases
Reintroduction of PR and push triggers for selected scenarios
Individual pipelines per image?
Testing and fixing of the functionalities of the containers, esp. for the supervisor and cron variants. Experience tells there might be regressions somewhere.
Testing and refinement of the instructions in the readme
Testing and documenting a successful migration path from the original standalone image to one of its new variants
Testing and documenting a successful migration path from the original source build to one of the new examples

Also new usages for the repository appear within reach:

Running containerised end-to-end integration tests for the containers, but also possibly for pretalx itself
Examples for using these images with other container schedulers, such as Nomad or Kubernetes

Ultimately, I will also raise conversations upstream about implementing Celery Beat, which would allow to get rid of the cron dependency entirely, and about a containerised development environment and why one might like that.

shoetten

Wow, what a huge PR! First of: sorry that i'm not active here. As the story often goes, i don't use pretalx anymore, so i'm not keeping up here and (as the state of this repo shows) i only maintain this on paper..

Nevertheless i took the time to look at your PR. All in all, i like a lot of your changes. Especially breaking up the container images in multiple ones, going more in the direction of one container, one process, as it should be. I left a few comments, but please, just take them as suggestions, nothing important.

I have two main suggestions:

Simplifying
As nice as having all the different variants is, i fear this might be way to much to maintain in the future. If you're up for the job: awesome! Otherwise i would suggest deleting some of them. E.g. do we need the ability to build from source? I know it would be nice to be able to run pretalx:main, but since @rixx wants to release more often, anyways, i don't know if its worth the extra trouble.

Readme

I think most people just want a quick, opinionated "how to run this in the simplest way on production" from the readme. Right now this seems kind of buried. Would you be up for writing a short "how to use in production" and put it up top?
I would also suggest moving everything related to building the containers and developing this repo out of the readme, e.g. in a docs folder.

As is said, please just take these points as suggestions. As these changes go in the right direction overall, i'm not objected to merging as is.

shoetten · 2024-05-25T18:17:18Z

context/base/Dockerfile.debian

+FROM ${BASE_IMAGE}:${BASE_TAG}
+
+# Install system dependencies
+RUN apt update && \


the modern way of doing this would be to use multi stage builds. then you can avoid the "install and clean" dance in one giant "RUN" command and copy only the bits over to the next stage that you need

shoetten · 2024-05-25T18:20:02Z

context/base/Dockerfile.debian

+                libmemcached-dev \
+                locales \
+                nodejs \
+                npm \


where do we need node and npm? in the source variant?

shoetten · 2024-05-25T18:39:02Z

legacy/README.md

@@ -0,0 +1,145 @@
+# Legacy docker-compose support
+
+This example demonstrates a setup that is compatible with the legacy version of Docker Compose.


Maybe make it more obvious that you mean docker-compose with the dash?

Suggested change

This example demonstrates a setup that is compatible with the legacy version of Docker Compose.

This example demonstrates a setup that is compatible with the legacy version of `docker-compose` v1.

shoetten · 2024-05-25T18:40:23Z

README.md

+docker compose -f compose.yml -f compose/local.yml -f compose/traefik.yml config
+```
+
+### Live


Suggested change

### Live

### Production

I feel the word "production" is more commonly used, than live?

shoetten · 2024-05-25T19:00:50Z

README.md

+
+This repository follows the Pretalx installation instructions as closely as possible.
+
+- [Installation — pretalx documentation](https://docs.pretalx.org/administrator/installation/)


this seems kind of misleading, since the official docs don't have anything to do with docker at all..

almereyda added 13 commits May 6, 2024 21:14

Remove Ansible 🏵

153d5be

Remove Submodule 🏵

de68785

Rename build workflow

6480799

Remove reverse proxy examples 🏵

add6812

Remove Supervisor configuration 🏵

4642939

Fix(CI): rename container image

5dd26a1

Chore: update gitignore

12996d3

Feat: Add LICENSE

a0e5449

Feat: Build container from release version, add cron, update entrypoint

8e50e02

Feat: Rework Compose manifest, add dotenv and nginx configuration

5904427

Feat: Add plugins compose + context to build image with plugins

efc7f81

Feat: Add compose for local development and Traefik deployment

a273bd5

Chore: relocate pretalx.cfg example

8991235

almereyda mentioned this pull request May 9, 2024

fix: pass environment variables down through supervisord #63

Open

Docs(README): adapt to build + compose changes

cdb9bc4

almereyda force-pushed the main branch from 6970a37 to cdb9bc4 Compare May 9, 2024 20:36

DASPRiD reviewed May 9, 2024

View reviewed changes

context/default/Dockerfile Outdated Show resolved Hide resolved

DASPRiD reviewed May 9, 2024

View reviewed changes

almereyda added 9 commits May 10, 2024 00:11

Feat: parametrise Dockerfile with build arguments

5c53801

Feat: split local into an independent build overlay

1f2ac77

Docs(README): adapt + simplify explanations, wording and typos

a00895e

Fix: crontab

62ed514

Fix(typing): reintroduce libmemcached-dev dependency

2be5f93

Chore: set Python version to 3.12

703579f

Chore: reduce compose.yml fragments syntax

81f4bf9

Docs(README): typos

ae6b659

partially reverts a00895e

Docs(README): corrections

76188b1

almereyda mentioned this pull request May 18, 2024

Locale only partially applied, mostly English fallback loaded pretalx/pretalx#1773

Closed

almereyda added 21 commits May 22, 2024 21:32

Chore(CI): Remove automatic pipeline triggers

0747a63

Feat(CI): add pipeline to build all images

146518d

Fix(CI): reduce number of inputs on workflow_dispatch

f0d2f73

Fix(CI): file input default value

87bc5e8

Fix(CI): Provide actual job names

8e1e016

Fix(CI): build dependent images from digest

f84ac9e

Fix(CI), build dependent images from digest, take II

0bce9b1

Fix(CI), build dependent images from digest, take III

df65a0f

Fix(CI): separate base and default manifests

aee0664

Chore: default to PIP user installation

ec5aa56

Chore: remove unused build stage names in single-stage image manifests

f6123c1

Fix(CI): scope BASE_* build arguments to pretalx, where semantically …

eca13c0

…appropriate

Fix(CI): do not scope BASE_* build arguments to pretalx, if not insta…

a5ce709

…lled yet

Fix(CI): Use expected artifact names

273140c

Chore: remove further unused build stage names

2f88e9f

Fix(CI): hard-code build contexts to remove ambiguity

8b24175

Fix(CI): add missing meta data of imported image to job environment

0d2630c

Feat(CI): introduce release trigger for building all images, prepare …

796a7a9

…ISO date versions

Feat(CI): Compatibility with triggers without inputs

8bea177

Docs(README): add base context

12a2d63

shoetten reviewed May 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: rework Container image, Compose manifest and CI #64

Feat: rework Container image, Compose manifest and CI #64

almereyda commented May 9, 2024 •

edited

almereyda commented May 9, 2024 •

edited

almereyda commented May 9, 2024

DASPRiD May 9, 2024

almereyda May 9, 2024 •

edited

DASPRiD commented May 9, 2024

almereyda commented May 13, 2024

rixx commented May 15, 2024

almereyda commented May 16, 2024

rixx commented May 17, 2024

almereyda commented May 17, 2024 •

edited

almereyda commented May 20, 2024 •

edited

almereyda commented May 23, 2024 •

edited

shoetten left a comment

shoetten May 25, 2024

shoetten May 25, 2024

shoetten May 25, 2024

shoetten May 25, 2024

shoetten May 25, 2024

		@@ -0,0 +1,145 @@
		# Legacy docker-compose support

		This example demonstrates a setup that is compatible with the legacy version of Docker Compose.


		This repository follows the Pretalx installation instructions as closely as possible.

		- [Installation — pretalx documentation](https://docs.pretalx.org/administrator/installation/)

Feat: rework Container image, Compose manifest and CI #64

Are you sure you want to change the base?

Feat: rework Container image, Compose manifest and CI #64

Conversation

almereyda commented May 9, 2024 • edited

almereyda commented May 9, 2024 • edited

almereyda commented May 9, 2024

DASPRiD May 9, 2024

Choose a reason for hiding this comment

almereyda May 9, 2024 • edited

Choose a reason for hiding this comment

DASPRiD commented May 9, 2024

almereyda commented May 13, 2024

rixx commented May 15, 2024

almereyda commented May 16, 2024

rixx commented May 17, 2024

almereyda commented May 17, 2024 • edited

almereyda commented May 20, 2024 • edited

almereyda commented May 23, 2024 • edited

shoetten left a comment

Choose a reason for hiding this comment

shoetten May 25, 2024

Choose a reason for hiding this comment

shoetten May 25, 2024

Choose a reason for hiding this comment

shoetten May 25, 2024

Choose a reason for hiding this comment

shoetten May 25, 2024

Choose a reason for hiding this comment

shoetten May 25, 2024

Choose a reason for hiding this comment

almereyda commented May 9, 2024 •

edited

almereyda commented May 9, 2024 •

edited

almereyda May 9, 2024 •

edited

almereyda commented May 17, 2024 •

edited

almereyda commented May 20, 2024 •

edited

almereyda commented May 23, 2024 •

edited