nixos/kanidm: add basic provisioning

[oddlama]

Description of changes

This PR adds provisioning of persons, groups and oauth2 systems to kanidm, and allows declarative provisioning of oauth2 basic secrets and admin/idm_admin account credentials.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[h7x4]

This is really cool. Looking forward to it!

[oddlama]

I've pushed some updates to this since kanidm is steadily approaching a stable release by now, and so I though this might be a good time to revisit this.

Changes:

• Removed jank. No more database editing, no more scripting using the kanidm CLI. I've written a new provisioning utility that uses the kanidm API instead of relying on the CLI util (which should not be used for scripting according to kanidm). This is a lot faster (30s to below 1s) and more controllable in the future.
• Since provisioning secrets is explicitly against kanidm's goal of being the only source of truth, this will not be supported upstream. The only remaining option thus was to patch kanidm. I've included two optional patches here, one allows setting the admin credential when recovering accounts and the other adds an endpoint to set oauth2 basic secrets. These patches are only required when any secret should be provisioned, the regular provisioning can be used without them. When a user tries to set a secret provisioning option, an assert automatically hints at the correct option which is services.kanidm.package = pkgs.kanidm.withSecretProvisioning;. This was chosen so the original kanidm package is not by default affected by any patches.
• Setting present = false; is no longer required by default (there now an option for this called autoRemove). The new utility can reliably track which entities it creates and thus remove them again if needed.
• Some new options can now be provisioned, like claim maps and PKCE disable.

I've chosen to go with person, group and oauth2 provisioning for now, which suffices for classical homelab SSO and OIDC use cases. In theory we can also have unix user/group stuff, ssh key and radius provisioning in the future, but cut the scope here for now.

The NixOS tests for provisioning are not as exhaustive as I'd like them to be, so I'll address this tomorrow. Feel free to leave your suggestions here :)

[oddlama]

Alright, I've added extensive tests now, this should be good to go. One thing I noticed is that orphaned claim maps cannot be reliably removed right now, since the required output has only been recently added to kanidm (2 weeks ago). So we need to wait for v1.1.0-rc.17 for that to work, but it doesn't impact any other functionality. I put a comment in the testing code so we can enable the test case once the next version is released.

[Flakebi]

Ran the test and tested locally, both works as expected.
Thanks, this is great!

[Flakebi]

This got some merge conflicts due to treewide doc changes

[oddlama]

Thanks for the info, i've rebased it on top of the latest master.

[erictapen]

My 5¢ here are that I'm highly skeptical of adding this kind of database provisioning to NixOS, especially with this requiring so many patches as upstream doesn't want to cover provisioning as a use case. IMO this would work better as a separate Flake.

[Flakebi]

Thanks for the rebase!

I have been using the user/group/oauth provisioning (without the patches) in the last month and I’m quite happy with how much easier everything got.
@erictapen, are you fine with adding the parts here that don’t require the patches (I think everything but secret provisioning)?

[erictapen]

@Flakebi Oh I'm fine with bringing this forward as is, iff people are enthusiastic about it. Just wanted to voice my skepticism.

[alexandru0-dev]

@oddlama love the work

About the patches rn can't be upstream as I've read already the issue that u made on the kanidm's repo.
So unless they change their minds or there is so much requests for this feature, those needs to be downstream.

Personally i think downstreaming those wouldn't be so much of a problem, tho improving clarity (using the description), warning about the fact that those patches are not associated with the kanidm project and any related issue about provisioning and patches to be reported to your repo for example.
(Better issue reporting and less burden for the kanidm team if issues are reported wrongfully there)

@erictapen @Flakebi tell me what you think about it.

[oddlama]

Updated to support kanidm 1.2.0. Beware that the tests can currently not run correctly because the CLI logout command is broken in kanidm 1.2.0 (refer to their bug tracker for more information). I've locally verified that the tests still work when executed manually. This should fix itself by waiting i guess.

[adamcstephens]

Are the lib changes necessary? They add a lot of churn to this PR just shifting things around.

[oddlama]

Are the lib changes necessary? They add a lot of churn to this PR just shifting things around.

That's just how I usually structure modules when they use a lot of lib, of course it's not necessary :)
I can revert back to lib.* everywhere if that makes it easier to digest.

Another thing: I'm not quite sure yet on how to handle is the update process for kanidm after this change. I would like this PR to not block the regular updating process for the unpatched kanidm in any way. But the patches are currently living in nixpkgs and the nixos tests will test both the unpatched and the patched variant.

If a future update to kanidm comes along, the tests cannot succeed until the patches are updated. Does anyone have an idea on how we can decouple the tests into kanidm tests and "patched" kanidm tests that can run independently?

[oddlama]

Re-added lib., added enableSecretProvisioning comment linking to upstream, removed accidential doCheck = false. Rebased on master.

[Flakebi]

Thanks for updating. Looks good to me, except for two unused options.

[oddlama]

If a future update to kanidm comes along, the tests cannot succeed until the patches are updated. Does anyone have an idea on how we can decouple the tests into kanidm tests and "patched" kanidm tests that can run independently?

Any opinions on this?