Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

chadfurman · 2024-04-27T14:52:10Z

This is an implementation of mailbox quotas using dovecot's quota capabilities. This pull request supersedes a previous pull request that contained some changes not pertinent to the main repository.

^ @jrsupplee

This PR is an update of #1568 ported to the latest version of MiaB.

TODO:

Move conf files to editorconf tool
- 15-mailboxes.conf
- 20-imap.conf
- 90-quota.conf
get confirmation that we're okay with how the quota column is being added in the setup script - is this maintainable?
get confirmation we're okay with recalculating quotas on setup -- seems fine to me?
Figure out why JRSupplee had his own bootstrap script and resolve any discrepencies -- I'm guessing it's largely related to the conf files but could also be initializing dovecot somehow tbd
get approval from MiaB team and get merged into mainline

chadfurman · 2024-04-27T15:23:51Z

Looks like 15-mailbox.conf is just relocated and the script is already updated so it's fine as-is.

chadfurman · 2024-04-27T15:41:06Z

Does not appear to be any reason why https://raw.githubusercontent.com/jrsupplee/mailinabox/master/setup/bootstrap.sh was being used instead of https://mailinabox.email/setup.sh?ping=1 -- my best guess is for reasons of pinning the version

chadfurman · 2024-04-27T16:58:02Z

.gitignore

@@ -5,5 +5,6 @@ tools/__pycache__/
 externals/
 .env
 .vagrant
+.idea/


small change happy to remove if you want

chadfurman · 2024-04-27T17:01:06Z

Woops looks like basing off main was a bad idea. Gonna redo this and base instead off of v68

chadfurman · 2024-04-27T22:58:22Z

So the conflict is trivial to resolve (just bring in the first line of the if block and also the elseif block) but when I do, it also pulls in the latest commits from main and it causes the setup to break (due to the no password provided bug) so I'm not resolving conflicts just yet.

chadfurman · 2024-04-27T23:27:29Z

strangely, I get a 500 error the first time I run setup/start.sh but the second time it works?

chadfurman · 2024-04-27T23:30:41Z

When it 500s after setting the password, it seems it fails to create the admin user.

Creating the admin user by hand seems to work?

ubuntu@ip-172-31-36-105:~/mailinabox$ sudo management/cli.py user make-admin 'me@agnai.guide'
sudo: unable to resolve host box.agnai.guide: No address associated with hostname
OK
ubuntu@ip-172-31-36-105:~/mailinabox$ sudo management/cli.py alias add administrator@agnai.guide me@agnai.guide
sudo: unable to resolve host box.agnai.guide: No address associated with hostname
alias added

chadfurman · 2024-05-05T02:45:40Z

Fixed the 500, it was related to missing the subprocess module. Fixed some conflicts with the spamassassin.sh script also

chadfurman · 2024-05-05T03:04:10Z

I've resolved the conflict -- the remote chadfurman/mailinabox has a somewhat stable and tested version of this tagged as v68a (i.e. git checkout v68a), but the head now lines up with mail-in-a-box/mailinabox to make this easier to merge / manage.

dms00 · 2024-05-06T16:51:30Z

management/cli.py

@@ -117,6 +123,14 @@ def setup_key_auth(mgmt_uri):
 			if "admin" in user['privileges']:
 				print(user['email'])

+elif sys.argv[1] == "user" and sys.argv[2] == "quota" and len(sys.argv) == 4:
+	# Set a user's quota


Is this comment correct? I don't understand how this sets a user's quota.

stylnchris · 2024-05-06T17:03:45Z

Not sure if it helps or not but I've installed this with the v68a 'tag' on my production MiaB server and can confirm it works.

Steps I've taken to install it:

ssh into your box -

cd /root/
mv mailinabox mailinabox.old
git clone https://github.com/chadfurman/mailinabox.git
cd mailinabox
git checkout v68a
sudo setup/start.sh

dms00 · 2024-05-06T17:04:41Z

setup/mail-users.sh

 	echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
 	echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path";
 	echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
+elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then
+  echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path;


You've opted to make the quota column type TEXT versus making it a numeric column. Just curious if there was a reason for this choice.

Pretty sure what is stored in sqlite is basically for the webui to 'display' with what is actually set with maildirsize file for dovecot.

Also don't forget this is based on proven code from @jrsupplee, so some of this code is obviously not written by Chad.

dms00 · 2024-05-06T17:11:22Z

setup/mail-users.sh

@@ -20,10 +20,12 @@ db_path=$STORAGE_ROOT/mail/users.sqlite
 # Create an empty database if it doesn't yet exist.
 if [ ! -f "$db_path" ]; then
 	echo "Creating new user database: $db_path";
-	echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 "$db_path";
+	echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '', quota TEXT NOT NULL DEFAULT '0');" | sqlite3 $db_path;


If I understand correctly, '0' means unlimited and non-zero represents the quota. But there's no symbolic representation for "system default' from what I can see. So for example, if I made the system default 1GB, then the users would all get their DB value set to 1GB. If I later were to increase the system default to 10GB, the existing users would stay at 1GB and not inherit the new default.

Correct me if I've overlooked something and that's not how it works, but assuming that description is correct, then I'm wondering if the users that are getting the "system default quota" should have a symbolic representation of this in the DB, so if the default were to be changed, these users inherit the new value automatically.

Quota is not applied, by default so it is basically set to 0. 0 represents "no quota or 'unlimited' quota" (obviously the limit is the size of the hard-drive for miab server.

creating a new user will give you the option to specify a quota with the new user.

here is an example of setting the quota:

Then enter the number and and click the "set quota" button

I think my confusion stems from the fact that 0 means unlimited, but I also see a system default-quota value that can be set, and I'm not sure what role that default value plays in this. How would I set a user to the "default-quota" value?

@dms00 - Will not having some of these answers prevent this from being merged? Again, code is based on original code from PR - #1568 created by @jrsupplee. Of which we don't have all the answers as to why things were done the original way. Perhaps this was meant for something that wasn't implemented. I do appreciate you reviewing this though. If its required for the PR to be completed we can try to figure out what John was doing.

I've tried to use it but when I set quota and try to send a mail that has attachments to exceed the quota it errors but mail still sent but it didn't saved as sent folder.. How can I prevent that?

So you installed the fork, you set a quota. Then you tried to send mail out of that mailbox in order to exceed the quota. It sends the mail, but doesnt save the mail in the sent box.

Am I understanding the above correctly?

You wouldn't prevent that, that is the expected result. Once a user hits a quota that is set, they need to either remove mail from their mailbox, sent folder, trash, etc or things become a bit unstable (by design) you wont be allowed to future things to disk as you exceed the quota.

How can you prevent it, give yourself more quota or delete mail.

Maybe I dont understand correctly though....

You understand correctly but I want to prevent that email sent. If quota is full user shouldn't be able to use it. And what can be do about 1 quota per domain? like example.com has 10GB total quota and every user in it like hi@example.com go@example.com has same quota based on domain but it will not affect to example.net?

Okay I understand what you would like to see, but I believe this is a limitation of Dovecot / Postfix.

chadfurman mentioned this pull request Apr 27, 2024

Mailbox Quotas #1568

Open

chadfurman commented Apr 27, 2024

View reviewed changes

.gitignore Outdated

@@ -5,5 +5,6 @@ tools/__pycache__/

externals/

.env

.vagrant

.idea/

Copy link

Author

chadfurman Apr 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small change happy to remove if you want

chadfurman force-pushed the master branch from bd84c50 to 8a0c716 Compare April 27, 2024 17:10

bringing in quota changes

1795f8a

chadfurman force-pushed the master branch 2 times, most recently from f82be1b to 1795f8a Compare April 27, 2024 22:54

fixing subprocess import

3970038

chadfurman force-pushed the master branch from f59b91c to 3970038 Compare May 4, 2024 18:27

chadfurman added 3 commits May 4, 2024 21:02

fixing imap sed script

173501e

fixing parens

51c4831

fixing imap sed script

0cd8e4d

Merge branch 'main' into master

0408ec3

dms00 reviewed May 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

chadfurman commented Apr 27, 2024 •

edited

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented May 5, 2024

chadfurman commented May 5, 2024

dms00 May 6, 2024

stylnchris commented May 6, 2024 •

edited

dms00 May 6, 2024

stylnchris May 6, 2024

dms00 May 6, 2024

stylnchris May 6, 2024 •

edited

stylnchris May 6, 2024

stylnchris May 6, 2024

dms00 May 6, 2024

stylnchris May 6, 2024

MrMirhan May 10, 2024

stylnchris May 10, 2024

MrMirhan May 10, 2024

stylnchris May 10, 2024

Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

Are you sure you want to change the base?

Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

Conversation

chadfurman commented Apr 27, 2024 • edited

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

Choose a reason for hiding this comment

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented Apr 27, 2024

chadfurman commented May 5, 2024

chadfurman commented May 5, 2024

Choose a reason for hiding this comment

stylnchris commented May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stylnchris May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chadfurman commented Apr 27, 2024 •

edited

stylnchris commented May 6, 2024 •

edited

stylnchris May 6, 2024 •

edited