mautrix-meta: init at 0.2.0

[Rutherther]

Description of changes

This PR adds new package mautrix-meta which is quite new Matrix bridge for both Facebook and Instagram - https://github.com/mautrix/meta, it aims to replace mautrix/facebook and mautrix/instagram. The PR also adds service for this package, based mostly on mautrix-whatsapp with some changes, such as making the service's package and dataDir configurable.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[Rutherther]

As someone pointed out in my other pr #293830 that changes mautrix-telegram service, the StateDirectory can be only under /var/lib. I have thus made the dataDir relative under /var/lib, and mentioned in the comment it cannot go elsewhere.

I will also add some tests based on the ones that are in mautrix-discord PR during the weekend.

Any better ideas about the dataDir are welcome. But I think it makes sense to be configurable.

[kevincox]

I tried this out and it is working other than the base module generating the registration info in a private directory. But I worked around this by managing my registration separately as described in a thread.

[Rutherther]

Thank you for the review! I will have a look on the things you mentioned.

[kevincox]

I think this is backwards. It copies from the registration file which is randomly generated to the settings file. This makes the configured value for the settings useless. (And contradicts the docs for environmentFile which imply that you can use it to configure these tokens)

[Rutherther]

I dont think this is true completely. The values are generated randomly only if the value is not configured in the config. If it is, the registration file will contain these. So its possible to first put in config values, then let it generate, and it will contain the ones you set, not random ones.

[kevincox]

I guess that is true but at least after the first generation they values will be ignored. So this results in pinning the values from the first configuration which is surprising.

[Rutherther]

I will solve this issue by regenerating the registration file if as_token and hs_token are present in the configuration.

[Rutherther]

I've changed the behavior so that the registration file gets as_token and hs_token replaced with the ones from settings, if they are set. And if not, the values from registration are used to allow random generation by the bridge as well.

It also looks like I was not correct when saying that the as_token, hs_token do not get generated if they are already set in the config. I am pretty sure that's the case for example for mautrix-telegram, but doesn't seem so for this bridge. So I've also made sure the options from config are going to be used if the registration is generated.

[Rutherther]

What do you mean by this?

I meant the mautrix-meta --generate-registration. At first I thought it would be great, but since I learned that it changes the values in config, including as_token and hs_token, I began to be afraid of what it could break.

I'm not 100% sure what you are describing. Are you saying just replace the built-in registration with our own script? Then extract the {as,hs}_token values and sub them into the config? I think that would work well.

i am suggesting doing approach like with settingsFileUnsubstituted, but for the registrationFile. And moreover, if the user does not configure the hs_token, as_token fields, they will be automatically generated - to still allow for approach where you don't want to use something like sops, and want to avoid having secrets in the store.

So there would be something like this to configure:

      registrationData = {
        id = cfg.settings.appservice.id;
        as_token = "$AS_TOKEN";
        hs_token = "$HS_TOKEN";

        namespaces = {
          users = [
            {
              exclusive = true;
              regex = escapeRegex "@${cfg.settings.appservice.bot_username}:${cfg.settings.homeserver.domain}";
            }
            {
              exclusive = true;
              regex = "@${puppetRegex}:${escapeRegex cfg.settings.homeserver.domain}";
            }
          ];
          aliases = [];
        };

        url = cfg.settings.appservice.address;
        sender_localpart = "mautrix-meta-sender";

        rate_limited = false;
        "de.sorunome.msc2409.push_ephemeral" = true;
        push_ephemeral = true;
      };
    };

Then it would be written into nix store by pkgs.formats.yaml.generate, and lastly, the registration service would envsubst this file into /var/lib/mautrix-meta-X/meta-registration.yaml. And in case AS_TOKEN, HS_TOKEN were missing, they would be generated. I am thinking of saving these into /var/lib/mautrix-meta-X/secrets, since both config and registration can get replaced and this seems like the easiest solution to make sure of not losing them.

I think that instead of using yq, we could stick to just envsubst even in the config.

[kevincox]

Personally I would rather just overwrite the {hs,as}_token in the file generated by mautrix-meta with jq/yq as it is more robust. We know exactly what keys should have which values so it seems easiest to use these tools that make the exact syntax-aware substitution rather than some text substitution.

If anything I would prefer to do everything this way. Instead of environmentFile I'd rather have something like secretSettings which is a JSON file that gets merged in. It is just a more precise operation. Especially if we just need these two tokens.

[Rutherther]

I've updated it. What do you think about it now?

[kevincox]

I won't be able to look again until tomorrow. I'll get back then.

[kevincox]

This looks good.

[Rutherther]

Okay so I've changed this to draft as there will be more changes needed. I have started with implementing changes you proposed. The current state is that the services are not starting properly. I will take a look during the weekend. Could you let me know if you think it's going on in the right direction?

I have also an issue with the default instances for facebook and instagram. It seems like I am overriding the default value even if I try to put in appy = lib.recursiveUpdate default and I am not really able to tell why...

[Rutherther]

@SuperSandro2000 Thank you for the suggestions. I have implemented them all. Any more suggestions are welcome.

[Rutherther]

I have solved the predefined instances. Now there are both facebook and instagram with predefined options, and they can be enabled. It doesn't suffice to only enable them though, as permissions and homeserver info have to be set. I think it will be better to force the user to configure these than to rely on the defaults.

[kevincox]

The best way to do this is probably to create a separate group that both services can be added to. Then the file can be owned by mautrix-meta but readable by the group.

[Rutherther]

It could stay readable by mautrix-meta-X since they are the user owner, no? So it would be enough to change group owner and give them read permissions. The problem I was facing that the chown was giving no permissions and I couldn't figure out why... that's why commented it out. Any idea? I suppose it cannot be because mautrix-meta-X is not member of that group, can it?

[kevincox]

Yes, mautrix-meta-X. I was just being lazy.

it would be enough to change group owner

I guess it would but I tend to avoid this because you can only have one group owner. If you create a specific group for this data you can assign any number of readers. But I guess multiple readers is probably not necessary for this data.

Any idea? I suppose it cannot be because mautrix-meta-X is not member of that group, can it?

Yeah maybe. I always struggle to remember how changing ownership works. Does chown log an error?

[Rutherther]

Yes it does, but only permission denied / operation not permitted. I can reproduce it and copy it again, I looked into this yesterday I think, so don't have the exact error on hand. But there really wasn't anything else than operation not permitted :( That made me think if maybe the systemd service cannot chown files, but I was not able to figure out why that would be.

[Rutherther]

Don't worry about this, seems it was really just because mautrix-meta was not member of that group. I've created mautrix-meta-registration group, and now chown is possible.

[Rutherther]

I've created mautrix-meta-registration group, and assigned that to both the mautrix-meta users, and matrix-synapse user. It seems to be working with synapse finally!

[Rutherther]

So there was actually a bug, since the group was created only if synapse is active, but it was used every time. I fixed this by including the group every time, even when synapse is deactivated. And synapse user is included in the service only if synapse is enabled.

[kevincox]

That makes sense. Even if the user isn't registering synapse they will likely want to access this file by some service. So always having the group sounds good.

[kevincox]

I've updated to this version and it seems to work.

The main difference is other than the comments left here I'm using mautrix/meta@61f9c39 as it seems to improve backfill.

I'm also still generating the registration externally for reasons described elsewhere.

[Rutherther]

@kevincox I have integrated some changes. Most importantly, I added two tests, one for postgres and one for sqlite. I will rewrite one of them to use multiple instances to test it works properly, as you are right I have not tested it with that concatMapAttrs function + mautrix-synapse. Permissions for synapse are now resolved.

I have made it so that there is one user per instance, but just one mautrix-meta group. There is also mautrix-meta-registration group, all mautrix-meta-X users are in it, and also matrix-synapse (if synapse is enabled)

I am thinking if it would make sense to also make integration with postgresql to avoid some manual configuration, like here: https://github.com/NixOS/nixpkgs/pull/296718/files#diff-328b9996eb3362e17270fe65d6ee118c45f1380fcc54ef3c946c8842c5e587e5R20

[kevincox]

I have made it so that there is one user per instance, but just one mautrix-meta group.

I think this is probably fine.

[kevincox]

I think this is good. I've even been able to switch to the generated registration (with the automatic hookup)

[kevincox]

Just for reference here is my config (slightly redacted):

{ config, pkgs, lib, ... }: {
	imports = [
		./kevincox-backup.nix
		./synapse.nix
		./tor.nix
	];

	services.mautrix-meta.instances.facebook = {
		enable = true;
		settings = {
			homeserver.domain = "bridges.example";
			homeserver.address = "http://[::1]:8008";

			bridge = {
				delivery_receipts = true;
				backfill = {
						history_fetch_pages = -1;
						catchup_fetch_pages = -1;
				};

				permissions = {
					"@me:users.example" = "admin";
				};

				provisioning.shared_secret = "disable";
			};

			logging = {
				min_level = "info";
				writers = [{
					type = "stdout";
					format = "json";
				}];
			};

			meta = {
				mode = "facebook-tor";
				proxy = "socks5://localhost:${toString config.services.tor.client.socksListenAddress.port}";
			};
		};
	};

	kevincox.backup.paths = [ config.services.mautrix-meta.instances.facebook.dataDir ];
}

[Rutherther]

@kevincox Thank you very much for all of the suggestions. It's been very helpful and I am now very happy with the module. It was my intention from the beginning to allow multiple instances, since the bridge supports facebook and instagram, I just didn't know how to approach that and needed those few pushes.

I tested last version via the tests included and also with synapse and conduit servers locally with logging in to instagram to make sure there are no issues on bridge <-> instagram side. Everything seems to be working.

I'd be willing to also include a test with conduit, although I thought that if these tests are going to be ran on hydra, it's going to take up time unnecesarily. And also most of the use cases for this module were exhausted by using synapse only.

Marking this PR as ready for review.

[kevincox]

Thanks for all of the work improving it. I think it looks great! I can merge this whenever you think this is ready. I don't expect many more reviewers to stop by, and we can always make changes in a followup if needed.

[kevincox]

Result of nixpkgs-review pr 296718 run on x86_64-linux 1

2 packages blacklisted:

• nixos-install-tools
• tests.nixos-functions.nixos-test

4 packages built:

• mautrix-meta
• tests.testers.nixosTest-example
• tests.testers.runNixOSTest-example
• tests.trivial-builders.references

[Rutherther]

I noticed I forgot some mkDefaults in the configuration for facebook and instagram instances. So added these

[Rutherther]

I've deployed to my server, there were some issues related to permissions and default port change, but that was just because I had previous instance that handled users differently (DynamicUser was true), other than that it was fine.

So this is ready from my side.

[kevincox]

Sorry, GitHub never emailed me these comments for some reason. Let's go 🚀! Thanks for the great patch.