V2 #83

sthapa · 2024-03-27T05:06:05Z

Add updates to allow docker-compose with dev configs to work

russtoku

OK to commit.

tyliec

I think alot of the code in this PR can be removed. Our build steps have changed quite a bit to match fragdenstaat_de after [this PR],(#78) in particular these can be removed:

froide/*
local_settings.py
settings.py

We can then rewrite the Dockerfile & docker-compose.local.yml` file to match the instructions here - with an edge case for somehow handling the first initial seeding of the database.

tyliec · 2024-03-27T09:01:49Z

froide/.babelrc

We don't need this froide directory for Dockerization (at least after #78 got merged in). Side note, but the froide directory exists under ./src/froide after a pip install -r requirements.txt

The froide directory is there to build the image. It's a bit messy but I basically copied the okde froide repo into the froide directory so that I could build a clean version of the latest froide code. I figured having that alongside our code would help us update the froide code to get the functionality that we need.

tyliec · 2024-03-27T09:14:03Z

settings.py

The structure of the settings file has changed in main to closely resemble the file structure over in fragdenstaat_de - check out uipa_org/settings/

I'll look at that.

tyliec · 2024-03-27T09:15:43Z

local_settings.py

This file can be removed - check out uipa_org/settings/

The local_settings.py and settings.py files are there to load into the image when it's built. Look at the Dockerfile to see where this comes into play.

tyliec · 2024-03-27T09:24:26Z

docker-compose.local.yml

+
+services:
+  app:
+    image: ssthapa/uipa_backend:0.9


I don't love having a separate image here - but it's fine. We should have instructions on how to switch this out/publish our own containers though if so

I should actually have a 0.8, it's 0.9 because I was using that for testing. I'll fix that. The image is generated from the Dockerfile in the top level directory. We can just change the tag to uipa_backend:x to use a locally built copy. We need to document this though.

Dockerfile

tyliec · 2024-03-27T09:37:58Z

Dockerfile

+#ARG INSTALL_PYTHON_VERSION=${INSTALL_PYTHON_VERSION:-3.8.12}
+#FROM python:${INSTALL_PYTHON_VERSION}-slim-bullseye AS base
+# FROM debian:bullseye-slim AS base
+FROM ssthapa/django5:0.1


Where is this coming from?

That's something I built. The source is at https://github.com/sthapa/django-images

tyliec · 2024-03-27T09:40:16Z

Dockerfile

+COPY pyproject.toml /srv/django/pyproject.toml
+
+RUN apk update && \
+    apk add rust cargo qpdf qpdf-dev poppler poppler-dev g++ gdal geos alpine-sdk \


Do we really need all of these dependencies? I don't recall ever installing rust or cargo...

yeah we do, the python ssl module are rust code and with alpine I think pip needs to download and build the python module instead of just installing the pre-compiled verison.

tyliec · 2024-03-27T09:42:18Z

Dockerfile

+RUN /root/.local/bin/poetry install
+
+
+COPY local_settings.py /srv/django/froide/local_settings.py


We won't need to do this anymore - a Dockerfile matching the steps of https://github.com/codewithaloha/uipa?tab=readme-ov-file#instructions-1 should do the trick (with something handling the edge case of the first initial seed of the data)

The settings file doesn't seem to pickup the elasticsearch host from env variables. So I need to have a copy of the settings that tells django to try to connect to es on the network instead of localhost. I need to debug this a bit more but I figured this works for now so people could use this rather than waiting longer.

tyliec · 2024-03-27T09:47:19Z

requirements.txt

This file probably doesn't need to be updated for Dockerization, and can also be removed

Yeah, we should remove it and use the pyproject.toml with poetry instead.

Deleted it, although it sounds like we still use it in our workflow based on last night's session

Yeah, agreed. 😿

We should probably change that, I think going with poetry or pyenv would be better than venv initially and then telling people to use the pyproject.toml when they want to do things for real.

tyliec · 2024-03-27T09:51:45Z

I wasn't able to get this PR working locally as is, is there a permission issue as well somewhere?:

➜  uipa git:(v2) ✗ docker-compose -f docker-compose.local.yml up --force-recreate
[+] Running 1/1
 ✘ app Error                                                                                                                                                          1.9s 
Error response from daemon: pull access denied for sthapa/uipa_backend, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

russtoku · 2024-03-27T23:56:01Z

I get the same error:

$ docker manifest inspect ssthapa/uipa_backend:0.9
no such manifest: docker.io/ssthapa/uipa_backend:0.9

However, an earlier version of ssthapa/uipa_backend does exist.

$ docker manifest inspect ssthapa/uipa_backend:0.8
{
	"schemaVersion": 2,
	"mediaType": "application/vnd.oci.image.manifest.v1+json",
	"config": {
		"mediaType": "application/vnd.oci.image.config.v1+json",
		"digest": "sha256:c92a2bac4d2c7b6428986b281b07a5be2989995068848b60535db73d02e5fa74",
		"size": 12096
	},
	"layers": [
...
	],
	"annotations": {
		"org.opencontainers.image.base.digest": "sha256:fde9b5a36f85656f9df800b174af67d49a2189b0e7fdc8c0c5e4e1416f403a01",
		"org.opencontainers.image.base.name": "docker.io/ssthapa/django5:0.1"
	}
}

russtoku · 2024-03-28T00:19:31Z

Hmm...things seem a bit complicated and opaque. Is there a need to use a special image (ssthapa/uipa_backend)?

sthapa · 2024-03-28T04:59:35Z

@tyliec you'll need to login to dockerhub to pull the image I think. docker made it more difficult to use without a login.

@russtoku we should at the baseline provide a baseline image that works. If devs can change the image reference and use a locally built version but it's a bit more cumbersome and may take 5-10 minutes to build initially. Also with the pre-built image, it makes building a local image much faster.

russtoku · 2024-03-29T04:08:10Z

After running docker-compose -f docker-compose.local.yml up, I get this when I visit http://localhost:8000/:

I have to disagree with the use of poetry and the removal of requirements.txt for the use case of a person wanting to help with the backend and the frontend and trying to get a local working environment. If they can't run the Django dev server locally and must use Docker containers, what do they do? They won't have the appropriate experience and skills to debug things to get it working. Also, once they get a working Django dev server in a container, would they know how to go about making changes and presenting those changes in a pull request?

While Docker containers can be a way to quickly get up to speed, not everyone wants to and is able to use Docker locally. While it may be OK to use poetry in non-package mode to install the dependencies, it's probably beyond the reach of new-comers who lack the experience with it.

I'm OK with adding docker-compose.local.yml and pyproject.toml at the top-level but not with forcing people to use poetry to install dependencies.

sthapa · 2024-04-02T04:05:32Z

@russtoku What were you doing when you got that error? Can you try it with the 0.9 image and the updated docker compose file.

I'm okay with sticking with the requirements.txt file if we have a good setup and documentation that lets people use it but with all due respect, the current requirements workflow is broken. All the code with aloha devs have had to spend multiple hours getting things running with it with varying degrees of success. Yen spent 2 hours on it with our guidance and had to stop due to an error that none of us have seen. And that's the second or third session that we've tried to help her get things running. I think the current situation is a huge blocker for any new devs. If there's a documentation or code fix that'll resolve this, lets get that in and use the requirements.txt but unless we have a reliable way of onboarding new devs, it's a huge problem for us.

Using containers takes a little change in your workflow but it's possible to work locally using containers without issues. PyCharm and VScode both allow for editing and hotloading code into a container and I've personally debugged multithreaded c++ servers running on containers with the ability to step through functions, update variables and even disassemble code. You can edit code locally and see changes run on a container with fairly minimal chantges to your workflow.

russtoku · 2024-04-03T21:59:21Z

After running docker-compose -f docker-compose.local.yml up --force-recreate, I get this error when I visit http://localhost:8000/:

This is a different error from my previous attempt. In both cases, after cloning the repo and checking out you PR/83 branch, I run the docker-compose command to start up the containers.

The first try used ssthapa/uipa_backend:0.8 because there was no access to 0.9. The second try used ssthapa/uipa_backend:0.9.

This second try indicates that the migrations weren't run to initialze the database before the Django dev server was started. This would suggest that the sequencing of steps are not quite there yet. Here's the output from the dev server container:

app-1            | django-configurations version 2.5.1, using configuration Dev                        
app-1            | Performing system checks...                                                         
app-1            |                                                                                     
app-1            | /root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/weasyprint/text/fonts.py:65: UserWarning: No fonts configured in FontConfig. Expect ugly output.       
app-1            |   warn('No fonts configured in FontConfig. Expect ugly output.')                    
app-1            | System check identified no issues (0 silenced).                                     
app-1            |                                                                                     
app-1            | You have 326 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): accesstoken, account, admin, auth, bounce, campaign, comments, contenttypes, django_celery_beat, django_comments, document, easy_thumbnails, filingcabinet, flatpages, foirequest, foirequestfollower, foisite, frontpage, georegion, guide, letter, mfa, oauth2_provider, organization, problem, proof, publicbody, sessions, sites, taggit, team, upload.                                  
app-1            | Run 'python manage.py migrate' to apply them.                                       
app-1            | April 03, 2024 - 23:14:11                                                           
app-1            | Django version 5.0.3, using settings 'froide.settings'                              
app-1            | Starting development server at http://0.0.0.0:8000/                                 
app-1            | Quit the server with CONTROL-C.

russtoku · 2024-04-03T22:12:20Z

I'm personally OK with whatever is decided; even removing requirements.txt. I'm trying to advocate for a better developer experience so that anyone is welcome to try their hand at improving UIPA.

sthapa · 2024-04-04T04:29:34Z

@russtoku The error you're getting is probably due to the migrations not being applied. The migrations get applied before the django container runs so you shouldn't see that. Can you post the full output of the docker-compose up invocation?

russtoku · 2024-04-04T16:00:51Z

Sorry, I don’t have access to my Macbook Air for three weeks.

russtoku · 2024-05-08T00:06:24Z

With commit 3faa89f, this is what I get when trying to start the containers:

(py310) uipa (83)$ git log --oneline -n 5
3faa89f9 (HEAD -> pr/83, origin/v2) WIP getting uipa running in container
62e1b149 Point to new image version, clean up env vars
87a68578 Fix image reference and cleanups
d3c09cbf Merge branch 'main' into v2
f32a335a Fix elasticsearch references

(py310) uipa (83)$ docker-compose -f docker-compose.local.yml up --force-recreate
WARN[0000] /Users/russ/Projects/Code_With_Aloha/pr-83/uipa/docker-compose.local.yml: `version` is obsolete 
[+] Running 1/1
 ✘ app Error manifest for ssthapa/uipa_backend:0.10 not foun...            6.2s 
Error response from daemon: manifest for ssthapa/uipa_backend:0.10 not found: manifest unknown: manifest unknown

sthapa and others added 10 commits March 4, 2024 17:37

Add updated docker/podman config, use latest froide

6b77037

Set db value in compose file

57369e2

Add froide source code

8d99e01

Update settings

b65b9f0

Update backend start script and docker compose file

5dff6f9

Update backend start script and docker compose file

4a66b45

Install npm/yarn in backend image

2347b7a

Merge remote-tracking branch 'origin/v2' into v2

62eb6ba

Fixes for various django image and frontend

d082ce6

Fix elasticsearch references

f32a335

sthapa requested a review from a team as a code owner March 27, 2024 05:06

Merge branch 'main' into v2

d3c09cb

sthapa assigned tyliec Mar 27, 2024

russtoku previously approved these changes Mar 27, 2024

View reviewed changes

tyliec requested changes Mar 27, 2024

View reviewed changes

tyliec reviewed Mar 27, 2024

View reviewed changes

Fix image reference and cleanups

87a6857

sthapa dismissed russtoku’s stale review via 87a6857 March 28, 2024 05:14

Point to new image version, clean up env vars

62e1b14

WIP getting uipa running in container

3faa89f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V2 #83

V2 #83

sthapa commented Mar 27, 2024

russtoku left a comment

tyliec left a comment

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

tyliec Mar 27, 2024

sthapa Mar 28, 2024

sthapa Mar 28, 2024

tyliec Mar 29, 2024

sthapa Mar 29, 2024

tyliec commented Mar 27, 2024

russtoku commented Mar 27, 2024

russtoku commented Mar 28, 2024

sthapa commented Mar 28, 2024

russtoku commented Mar 29, 2024

sthapa commented Apr 2, 2024

russtoku commented Apr 3, 2024

russtoku commented Apr 3, 2024

sthapa commented Apr 4, 2024

russtoku commented Apr 4, 2024

russtoku commented May 8, 2024

		RUN /root/.local/bin/poetry install


		COPY local_settings.py /srv/django/froide/local_settings.py

V2 #83

Are you sure you want to change the base?

V2 #83

Conversation

sthapa commented Mar 27, 2024

russtoku left a comment

Choose a reason for hiding this comment

tyliec left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tyliec commented Mar 27, 2024

russtoku commented Mar 27, 2024

russtoku commented Mar 28, 2024

sthapa commented Mar 28, 2024

russtoku commented Mar 29, 2024

sthapa commented Apr 2, 2024

russtoku commented Apr 3, 2024

russtoku commented Apr 3, 2024

sthapa commented Apr 4, 2024

russtoku commented Apr 4, 2024

russtoku commented May 8, 2024