Docker internal changes

[sergiolaverde0]

Hi, I have created this to have a place where to discuss the changes I contribute to, since we have been misusing the issue tracker for that despite many of our exchanges not being proper bug reports.

For the upcoming 0.7 release, my tasks so far are:

• Finish the docker versioning changes, which should be fairly simple once I get down to it Done at feat: change the action from build on commit to main to build on release #124.
• Remove the JavaScript unneeded code from the final image Done at feat: reduce image size using a multi stage build #105.
• Remove possible unneeded Python code Not possible because of external dependencies.

Should anything more be proposed for 0.7, it goes here, and if something happens internally without affecting the end users, it goes here too and not in the issues since that would not be a bug but a part of the development process.

[simjanos-dev]

I think this is a great idea. Thank you! Should we close #59?

I may have found something. Docker desktop says, that this layer is 978.22 MB.

Command:

RUN /bin/sh -c python3 -m spacy download ja_core_news_sm && python3 -m spacy download ko_core_news_sm && python3 -m spacy download de_core_news_sm && python3 -m spacy download nb_core_news_sm && python3 -m spacy download es_core_news_sm && python3 -m spacy download zh_core_web_sm && python3 -m spacy download nl_core_news_sm && python3 -m spacy download fi_core_news_sm && python3 -m spacy download fr_core_news_sm && python3 -m spacy download it_core_news_sm && python3 -m spacy download ru_core_news_sm && python3 -m spacy download sv_core_news_sm && python3 -m spacy download uk_core_news_sm && python3 -m spacy download xx_ent_wiki_sm # buildkit

However, I looked at all the packages with dependencies, based on the spacy documentation's Dependencies section, and it does not add up.

ja_core_news_sm 11MB
ko_core_news_sm 13MB
de_core_news_sm 13MB
nb_core_news_sm 11MB
es_core_news_sm 12MB
zh_core_web_sm 46MB
nl_core_news_sm 12MB
fi_core_news_sm 13MB
fr_core_news_sm 15MB
it_core_news_sm 12MB
ru_core_news_sm 14MB
sv_core_news_sm 12MB
uk_core_news_sm 14MB
xx_ent_wiki_sm 10MB

SudachiPy ~3MB
mecab-ko ~1MB
mecab-ko-dic ~33MB
natto-py ~1MB
pymorphy3 ~1MB
pymorphy3-dicts-uk ~9MB
jieba ~20MB
spacy-pkuseg ~3MB

All added together: 279MB

Is it 3 times larger because the download size is compressed, or is there something to save? I found this, maybe it would save some space.

[simjanos-dev]

I have found a way to check if a language is installed, get a return value of 0 if it is and of 1 if it is not.
So we should have 3 functions in python: install, uninstall all and is language installed.

If it's less than 5MB, I think we can keep them in the docker image. I didn't know how large they are. Manga OCR will be large probably later on, so we will still need a system for it.

It is fine if you want to take it easier. It is Sunday and not even half of the month when our release is planned to the end of it.

Actually, I've been testing it for a few days, and found no problem. Things missing from v0.7 are:

• Merging PR from yesterday.
• Reworking readme a bit.
• Updating the overview website.
• Version numbers for docker.
• I may still add the website import if I'll feel like it.

I'll definitely slow down a bit. If I didn't make linguacafe public, It would have taken me over half a year to add all these stuff instead of a month.

[sergiolaverde0]

What setup do you use to test the backend changes? I have been thinking of doing the changes on the python container but I'm not very familiar with backend and the container does not quite expose any ports for me to route from to test it all works fine.

[simjanos-dev]

I wanted to write it into contributing.md, just haven't gotten to it yet.

I use the docker-compose-dev.yml in the dev branch. It's basically the old system that I had before you worked on it, without any copy command. I think you can create the same dev environment I use with this command:

docker compose -f ./docker-compose-dev.yml up -d --force-recreate

I forgot to remove debug mode from tokenizer.py, so the python service should reload and update if you modify the script.

I rewrote a lot of Laravel code on the current dev branch, it is not functioning 100% currently. I recommend use the main branch, or the commit on dev branch before I started working on the VocabularyController.

[sergiolaverde0]

Sorry, I didn't explain myself properly. I mean I am not sure how to test the API actually works because the Python container does not expose any ports so I don't know where to send the requests in the first place. For the Laravel one I could send GET requests to localhost:9191/path/to/be/tested and check I get the correct response, but for the Python container I'm not sure how to comunicate with it.

[simjanos-dev]

Oh, I understand it now.

When I write a python function, I just also write a laravel one, and use that to call the python function. You can find examples in TextBlock.php model's tokenizer function. You can reach other containers by using their name as domain name.

You could also expose the python port as well in the dev environment, and use some tool to make the calls directly. Or make them get calls temporarily, and open them in your browser.

For testing laravel url-s, I wrote a small tool to send requests, you can find it on the /dev url on the dev branch.

[simjanos-dev]

Maybe we could also merge the two containers together? The only benefit of having them separated is that Python does the majority of CPU intensive work, so it would be easy to have multiple python servers and one php. But I don't think it will ever have so many users that it will matter, and we can always separate them again.

[sergiolaverde0]

Yes, that is also a possibility. We could simply merge both dockerfiles into one and use the Python one as a step and then copy the binaries installed by pip into the final image (just like Node in the last PR) and then introduce changes as needed to ensure the process starts with the container (maybe adding it to entrypoint.sh?).

[simjanos-dev]

I will add these 2 libraries to python for v0.7:
pysub-parser
newspaper3k

They don't seem to take up a large space.

Edit: I was wrong, it did go up to 1.37 GB, I think it was 1.29GB before. I think it will be necessary to have an image where we allow enabling languages in a .env file, and only install missing python libraries at startup that the enabled languages need. Otherwise the python library will go over 2GB if I add other language models, other tokenizers for uncommon languages and OCR for manga reader.

It's impossible to keep the image size low, and add all the features at the same time. Will get even worse if generative AI gets better, text to speech is already seems to getting to an acceptable helpful level. I'm not sure what is the solution. I also don't think that I want to make users download and/or install these large libraries, if they only want the basic features.

(These are not important in the next few weeks/months, just some thoughts.)

[sergiolaverde0]

OCR will be a massive size increase, which reminds me of what other projects such as Stirling-PDF do by offering different images with either minimal or complete features, and sometimes points in between.

There is also the madness that is Rocker-2 which has like a dozen images built from another one, because some of them take over 4GB.

Manga OCR could be split together with the Japanese and it's 300MB of dependencies when the time comes.

None of this will be ready for 0.7, but is good to discuss it with time since is not an easy problem.

[sergiolaverde0]

On the topic of future possible features, I have been thinking if adding a RSS feed as a "book" where each entry in the feed is a "chapter" would be fitting since part of the recommended material are news feeds.

FreshRSS is less than 80MB compressed and is made in PHP so it makes for a good example.

[sergiolaverde0]

At some point some mad lad will ask to import a whole Calibre library, but I recommend against that for the foreseeable future. That would 100% need the queue, among other things, and maybe integration with calibre-web similar to the one with Anki.

[simjanos-dev]

I will try to keep the API-s and integrations somewhat limited. At a point it would get hard to maintain if I add support for every niche need.

I will add some form of mass import with the already existing import options though. I want to make myself a huge package of German or Spanish reading and listening material, and import it in LinguaCafe when it's more developed. But don't want store them imported while developing it this actively.

By the way, I will need to write a user manual. Importing large amount of text that you don't use is inadvisable, because whenever you save or delete a new phrase, it will has to be indexed in every book. I have over 700 phrases in Japanese, and over 4 books imported, and I've noticed some slowness when adding a new phrase (I've had a saving spinner for it, but removed it and do it in the background now).

[simjanos-dev]

I feel a tiny bit burned out, and changed my mind. I'll leave website and RSS import for v0.8.

[sergiolaverde0]

And that's fine. Current status on dev branch is something to be proud already, and I know people appreciate it.

[simjanos-dev]

At some point some mad lad will ask to import a whole Calibre library, but I recommend against that for the foreseeable future.

It really did happen. But I think it is a good idea, it would use a common API for book related software. I don't think I will add a mass import feature any time soon though.

[simjanos-dev]

Where should I create a "roadmap" for each update? I want to keep a list of the planned and finished changes for the next update.

I know about github projects, but there are changes that are not a separate github issue. Is it okay to do in the discussions section for each version?

[sergiolaverde0]

GitHub has something like called milestones (here an example) to keep track of the planned features for each tag however I'm not familiar with them.

I suggest to create them to discuss the details regarding the code itself and mark them as open or closed, but keep the discussion about things not directly related to code in the discussions section, such as what features to push on each version, potential release dates, distributing responsibilities between us and so on.

[simjanos-dev]

I think we should rename this and remove v0.7 from it, and just keep this discussion for any docker change in the future.

[simjanos-dev]

I may have asked this already, but I forgot and can't find it. Can I delete the docker-compose.yml file from the main branch? Someone tried to use it when v0.6 got released.

[sergiolaverde0]

Yes, it was superseded when we created the deploy branch.

[simjanos-dev]

There is an issue with the install process on Mac, it is not directly related to docker docker, but I thought I mention it here as well: #152. I'll try to find a fix for it.

I think we should also replace the Mac uncomment method with an environmental variable if possible.

I've been sick and working on a lot of refactoring in the last 2 weeks. I've just started working on features and bug fixes for the next update.

I'm planning on working on these:

• A user manual
• Word import from csv on the vocabulary page
• Website import
• RSS feed
• DeepL context
• Adding more DeepL target languages.
• Adding support for more dictionaries, including ones with non-english target languages.
• Smaller changes and bugfixes

I do not have any timeline for them. Also I may cut it into multiple updates.

@sergiolaverde0 Hope you are doing well!

[sergiolaverde0]

I've just remembered that before you started working on this issue I suggested adding a .env file, and make it auto install it at startup, and you said that it is not a good practice to install things inside the docker image after it was created and it slows down startup. But since we have decided doing it, what do you think of this solution:

We could create a bash file for installing languages, so the admin could just run it from the terminal after linguacafe is installed for the first time. It could still be stored in the storage directory, so it won't get deleted after a new image is pulled(I've realized we will also have to test this).

I somehow managed not to read this the first time. I think we can leverage our current solution by adding some environment variable(s) to store the languages that must be ensured to be installed always, then have the PHP container read it, compare it with the list returned by the Python container, and then install any missing languages one by one.

Shouldn't be too hard I think, reuses code and should be more robust than Bash.

[sergiolaverde0]

Done testing the five non-default languages.

Korean had issues the first time where the first chapter would be added but then the stats of the book would attempt to refresh seemingly forever, but after deleting the book and creating it all over again it worked perfectly fine, so I might attempt to reproduce it later to pin a possible cause. Korean was a bit underwhelming with its results since many tokens would not be available in the wiktionary file we provide and I suspect 제23조의 is not supposed to be a single token but somehow split but that is on the tokenizer and not Linguacafe.

Otherwise all five languages allowed me to create a book, import a couple of paragraphs of the Wikipedia page of the Universal Declaration of Human Rights, and check the definition and pronunciation of the tokens recognized. I also noticed I didn't know how to spell "Ukrainian" but I will amend the commit soon enough. Once that point gets addressed it will be ready for you to test.

While doing all that testing I found some small miscellaneous details about UI which you might, or might not, want to take into consideration. I will probably throw them all into a single issue despite not being really related to each other, but let me know if you prefer me doing it another way.

[simjanos-dev]

then have the PHP container read it, compare it with the list returned by the Python container, and then install any missing languages one by one.

I was suggesting this to make it simpler, and remove possible issues due to installing packages while the python script is running, or communication between many components, because I'm really worried about things going wrong. I think I would rather stick with the original UI plan than making it a background process with the php server.

Korean had issues the first time where the first chapter would be added but then the stats of the book would attempt to refresh seemingly forever, but after deleting the book and creating it all over again it worked perfectly fine

If you want to test if importing is working as intended, and you delete all books, I would suggest to also empty the unique_words table in the database for that language. It contains every unique word that has been imported, and if something fails for the first time, but works for the second, it can be a problem related to this (but unlikely).

Once that point gets addressed it will be ready for you to test.

I'll write the UI and will test it for a while, then I'll make a clean install and test it again. But my laptop is REALLY slow, so I would appreciate if you would also do a thorough final test after the UI is complete.

It is a critical part of the software, but it also won't cause large problems if it fails, because it would just return an error if someone tries to import something in a language that has not been installed yet. I don't think it can corrupt any data.

While doing all that testing I found some small miscellaneous details about UI which you might, or might not, want to take into consideration. I will probably throw them all into a single issue despite not being really related to each other, but let me know if you prefer me doing it another way.

It is fine in a single issue.

Thank you so much for working on this, I really appreciate it!

[sergiolaverde0]

I'm not sure I understood your proposal. I though you were asking for a script that would be copied inside one of the images, but:

remove possible issues due to installing packages while the python script is running

If the script is tied to the containers then it will run while the python script is running either way. Making it a separate bash script would only make it a separate process that runs parallel to the server which might actually make things worse since neither Python nor PHP would be informed of what is happening in said script.

I would rather stick with the original UI plan than making it a background process with the php server

And similarly, the bash script would be a background process all the same, so this would be as true.

So I think I misunderstood you and what you meant was a script to be shipped in the deploy branch instead of the actual images so that users could run it after cloning the branch and install the language models into the empty folders that were cloned too.

If that was indeed your proposal, then I don't think that is a good idea. If distributed this way the script would run on the host OS and any bug that happens would risk damaging the host instead of simply messing with a container that can easily repaired by restarting it; I'm certain people are tolerant to our little mishaps that erase their data because they only ever affect the software we provide and nothing else. Plus there is the issue of different systems having different versions of python/pip, and Windows support becoming even more spotty.

[simjanos-dev]

And similarly, the bash script would be a background process all the same, so this would be as true.

If the script is tied to the containers then it will run while the python script is running either way. Making it a separate bash script would only make it a separate process that runs parallel to the server which might actually make things worse since neither Python nor PHP would be informed of what is happening in said script.

I was asking for a script that would be inside the image, and your points are valid. You are right, I haven't thought this through.

[simjanos-dev]

Minor change related to docker image. Someone requested fonts for every language, so I'll write a font management system. I'll create a fonts directory inside the storage directory, and move the two default Japanese and Chinese fonts from the docker image to the deploy branch. Probably in v0.10.

[simjanos-dev]

Someone tried to update to v0.8 on MacOS without commenting the line, and they got this error on apple silicon processor:

I was surprised the git command worked. I've told them they have to uncomment the last line in docker-compose.yml. They did and run it again. After that the terminal did not answer for a while, then he got this error message:

I had no idea what it is. Someone in an other github project wrote that running it again fixed the issue. I told them to try it. After that linguacafe worked, he had his database deleted.

@sergiolaverde0 Any idea what could have happened? The only thing I can think of is that MySQL error. But I do not have any idea what exactly happened. Added an additional note to create a backup to the github release.

[sergiolaverde0]

I was surprised the git command worked

I tested it and it kept my changes after pulling, no warnings at all, since we haven't touched the file either so no conflict would happen. What I don't get is the error they got since MySQL does offer images for Arm64v8, and that same image should be pulled by all other Mac users, and be present already on their storage since it hasn't changed ever since we moved from 5.x months ago.

I had no idea what it is. Someone in an other github project wrote that running it again fixed the issue. I told them to try it. After that linguacafe worked, he had his database deleted.

My search pointed it to being an error on the git pull itself due to curl failing when reusing a connection, I assume when doing the second pull. Whatever happened to the database, this wasn't the cause.

Are we certain the database was deleted? I can understand it being corrupted, but the mounted volume shouldn't have its contents deleted. I would like to get a copy of the docker-compose the user had at the end of this odyssey, to check if there is anything wrong with it.

In any case this reminded you asked me to move away from the comment method and instead use a variable from the .env and I never did it, PR in a couple minutes.

[simjanos-dev]

Are we certain the database was deleted?

I'm not sure. He had no users in the database at least, because he was prompted to create the first user. He is not very familiar with the terminal, I didn't want to press for more details. He said he didn't have much data, so it's not a huge deal, but I still feel really bad.

I'm not sure what happened, I really don't think it was something with linguacafe, because I can't think of any significant change in the update that could cause this. 15 people updated the latest version, heard of no other problems.

In any case this reminded you asked me to move away from the comment method and instead use a variable from the .env and I never did it, PR in a couple minutes.

Thank you so much, I really appreciate it, especially making it so fast. This will certainly help some people. I also added a suggestion before I published it in large font on discord, reddit and the release notes to make a backup before updating.

(The chmod problem on mac was also solved. It was a problem with the argument order, for some reason Mac didn't support it.)

[simjanos-dev]

There was a comment thread below my reddit post about the docker process. I'm not sure what to think of it. I know the process is not conventional, but it is a simple step-by-step process to install and update.

I think it could be improved if we create the empty folders with the software after the storage directory was mounted. And copy the default files from inside the image to the mounted directory. After that it could be a simple docker-compose.yml file, and users would have to create the two directories, and it would be more similar to how other dockerized software usually works.

But I don't think we should do it. It can introduce problems, and I don't think there's anything wrong with the current install/update process itself, it's just different a little to what people are used to.

What do you think?

[sergiolaverde0]

DockerHub might provide better visibility, but not by much. I think the user last heard of the project back when you had to clone the repo and build the image yourself, hence the confusion.

I think it could be improved if we create the empty folders with the software after the storage directory was mounted

I tried my best at accomplishing this but failed to do so. The two issues we encountered are that users need root permissions to copy the dictionary files to a folder created by rootful docker (and rootless docker/podman have some rough edges) which is not ideal, and that Laravel can't write its logs to the mounted volume for the same reason since it runs as its own user instead of the user of the container. That's why we must do chmodon the whole thing.

I have had on mind asking you to (I am not sure I could do it myself) add a way to import dictionaries like we import epub books since that would solve the first issue as no files would have to be manually placed in the volume, but was waiting for development to settle down a bit more. For the second I looked at some selfhosted projects that use Laravel too and the only two I could find had similar issues with persistence.

I think achieving that would make deployment a breeze, and would enable Linguacafe (once it gets more stable) to be added to "stores" that offer single click installs of various selfhosted software like Cosmos and CasaOS since the current form is not very friendly to them.

And yet, I think there are more important points to address right now, as this smaller user base is easier to handle when we break something during development.

[simjanos-dev]

(Sorry, I'm a bit slow today, might overlook some simple stuff that is obvious.)

The two issues we encountered

I'm not sure if this is the exact same issue, but the temp folder also needs writing permissions, and it is shared between the two containers. Which also could be merged into one.

Cosmos and CasaOS

I've heard about the first, and was planning on adding linguacafe to it, but didn't have the time yet to look into it. I think it would be a good idea. It also sounds very useful for people not used to docker.

So, having a mounted storage is not an issue, the problem is that users should not copy files into it? I could replace it with a file upload. There will be also manga files(I think they can be several 100MB), other images files, audio and font files in the future which will be stored in the storage directory. Personally I prefer copying, but I think it is possible to make all these file upload based.

And yet, I think there are more important points to address right now, as this smaller user base is easier to handle when we break something during development.

I agree. I think it will take several years before linguacafe will be in a state where every feature people want will be implemented. I think I'll try focusing on the base reading experience and stability first.

By the way, speaking of stability. I recently I had a great idea: I think I should remove thousands of binded Vue events and classees (10 on every word) from the text reader, it might improve the performance a lot.

[sergiolaverde0]

I'm not sure if this is the exact same issue, but the temp folder also needs writing permissions

Ah right, this is a third issue I also forgot about the first time I touched the volumes and silently broke certain features. What a memory. Yes, it is also due to containers not being able to write into the volumes of a different one for security reasons.

So, having a mounted storage is not an issue, the problem is that users should not copy files into it?

Basically. An user that copies files needs permissions on the storage, an user that uploads files just needs permissions on the app.

There will be also manga files(I think they can be several 100MB)

I checked my own (translated) manga files and a chapter of is usually below 15Mb and almost always under 40Mb, while the one series I have compiled in tomes all are under 60Mb. Of course there can be lots of variation, but with this being selfhosted software I dare saying those sizes are reasonable.

[simjanos-dev]

When I go to the webserver package on github, I see this:

Is that third one supposed to be there?

[sergiolaverde0]

Seems to be a bug that got introduced at some point. Some of the older images were not affected, and it seems to be benign so no action from our side should be needed.

[simjanos-dev]

I was thinking about adding a backup function to the UI, since there are some users who have difficulties with the terminal, and I think even other people would be more inclined to click on a button compared to doing it manually.

I found that I can use mysqldump remotely:

mysqldump --host 127.0.0.1 --port 3307 the_database > some-host-file.sql

And then I think I could create a zip file from the .sql file and the storage folder. I think I could even write an import function for it.

This would require mysqldump command to be installed on the php container. I haven't checked yet how large is it. Do you think it is a good solution?

[sergiolaverde0]

I like the idea of dumping the database from the GUI since it does look a bit daunting, but I'm unsure about zipping together files that belong in different locations afterwards. You will need to be extremely clear on the instructions you provide for users to handle the import themselves since some might try to unzip all content in a single location and call it a day.

[simjanos-dev]

You will need to be extremely clear on the instructions you provide for users to handle the import themselves since some might try to unzip all content in a single location and call it a day.

I would make a UI for uploading the zip as well. I would unpack it, replace the storage folder, and import the .sql file. But I might just start with the database only, because it is much more important, easier, and currently the only problem with losing the storage folder is losing the book images, which can be corrected.

[sergiolaverde0]

Restoring the backup also introduces some edge cases you will need to take care of:

• If backup is imported after creating an account, handle the case where the user existing in the backup is different from the existing user; even more edge case where users share email but have different passwords. Alternatively, don't backup user details so that whatever "you" import is taken as "yours".
• You could allow a backup to be imported when you are prompted to create a new user when there are no existing users.

If the option to have multiple users is still planned to be added, we will need to implement this in a way that is compatible with them so that backwards compatibility is kept and refactoring is reduced to a minimum.

[simjanos-dev]

I was thinking about just doing a complete database export and import. So if someone restores the backup, all the current data would be lost. This would be just a UI for the process that is already described in the user manual.

I also do want to add a feature to export and import all of a single user's data in the future, but that will take a lot of work.

[sergiolaverde0]

Yes, sounds good.

[simjanos-dev]

This is so strange. There is quite a large difference between the php server image, and the python server image download counts.

I've heard no complains about anything docker related, so I don't think there's anything wrong.

[sergiolaverde0]

Oddly enough, for v0.7 it goes in one direction but for v0.8 and v0.9 it is the opposite. Even more, it also happens with the dev image.

Have there actually been changes to both images in all releases? Because if there was one where a container was identical to the previous release, docker is smart enough not to download it and just slaps the new tag on top of the old image. Then this creates a cascade effect.

[simjanos-dev]

I don't remember honestly, but I think in most of the releases they were both changed.

[simjanos-dev]

I've experimented a little with SQLite, but could not get it working. All the migrations are broken. SQLite uses different collations, which I will have to understand and test.

NativePHP seem to be progressing a bit. I won't have time to do it in the next year, but if it will work well, It could be used to package linguacafe as a standalone software by dropping languages that does not have space separated words, and a few features that use the python service.