Is it possible to define group-based access to a minio bucket?

[Diddy42]

Hi, great project!

Let's say I defined an additional bucket datasets with the value deploykf_opt.deploykf_minio.buckets.

I also have a group of users called machine-learning, defined with the value deploykf_core.deploykf_profiles_generator.groups.

Can I set some value so that the users of the group machine-learning have access to the bucket datasets?

Unless I'm missing something about minio policies, this cannot be done through deploykf_opt.deploykf_minio.policies, but I'd be happy to be proven wrong ^^

[thesuperzapper]

@Diddy42 First, I want to say that the embedded MinIO is NOT currently intended for production usage, it's meant to allow users to test deployKF anywhere, before they connect to a more robust S3-like object store.

(That is NOT to say you can't set up your own MinIO Object Store, and connect deployKF with it)

---

While it's a bit dangerous to encourage usage of the embedded MinIO, I am open to accepting a contribution that lets you define extraPolicyStatements, as we don't currently provide a way to extend the policies we automatically generate for each user defined in deploykf_core.deploykf_profiles_generator.users.

For example, we could add a value like deploykf_opt.deploykf_minio.extraPolicyStatements:

deploykf_opt:
  deploykf_minio:

    ## a list of extra policy statements to add to the default policy
    ## generated for each user in `deploykf_core.deploykf_profiles_generator.users`
    extraPolicyStatements:

      ## additional policy statements for 'user-1' 
      - user: user-1
        statements:
          ## allow 'user-1' to see MY_BUCKET_NAME
          - Effect: Allow
            Action:
              - s3:GetBucketLocation
              - s3:ListBucket
            Resource:
              - arn:aws:s3:::MY_BUCKET_NAME

      ## additional policy statements for each member of 'team-1' group
      - group: team-1
        statements:
          ## allow members of 'team-1' to see MY_BUCKET_NAME
          - Effect: Allow
            Action:
              - s3:GetBucketLocation
              - s3:ListBucket
            Resource:
              - arn:aws:s3:::MY_BUCKET_NAME
          
          ## allow members of 'team-1' to read/write under MY_BUCKET_NAME/some-prefix/*
          - Effect: Allow
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:DeleteObject
            Resource:
              - arn:aws:s3:::MY_BUCKET_NAME/some-prefix/*

The way this would be implemented would be by extending the automatically generated deploykf_opt.deploykf_minio.policies which are defined here:

deployKF/generator/templates/manifests/deploykf-opt/deploykf-minio/values.yaml

Lines 210 to 240 in 80b6483

	## minio access policies
	## - these policies are created and/or updated by a post-install job
	##
	{{<- $policies := .Values.deploykf_opt.deploykf_minio.policies | default coll.Slice >}}
	{{<- if and .Values.kubeflow_tools.pipelines.enabled (not .Values.kubeflow_tools.pipelines.objectStore.useExternal) >}}
	{{<- if eq .Values.deploykf_opt.deploykf_minio.identity.openid.policyClaim "email" >}}
	{{<- $users_id_mapping := tmpl.Exec "runtime/deploykf_profiles__users_id_mapping_json" | json >}}
	{{<- $users_profiles_access_mapping := tmpl.Exec "runtime/deploykf_profiles__users_profiles_access_mapping_json" | json >}}
	{{<- $bucket_name := .Values.kubeflow_tools.pipelines.bucket.name >}}
	{{<- range $user_id, $profiles_access_mapping := $users_profiles_access_mapping >}}
	{{<- /* get the minio policy name */ ->}}
	{{<- $user := index $users_id_mapping $user_id >}}
	{{<- $policy_name := $user.email >}}
	{{<- /* get the minio policy document */ ->}}
	{{<- $edit_profiles := coll.Slice >}}
	{{<- $view_profiles := coll.Slice >}}
	{{<- range $profile_name, $user_access := $profiles_access_mapping >}}
	{{<- if eq $user_access.role "edit" >}}
	{{<- $edit_profiles = $edit_profiles | append $profile_name >}}
	{{<- else if eq $user_access.role "view" >}}
	{{<- $view_profiles = $view_profiles | append $profile_name >}}
	{{<- end >}}
	{{<- end >}}
	{{<- $policy := tmpl.Exec "kubeflow_pipelines.object_store.user.minio_policy" (dict "edit_profiles" $edit_profiles "view_profiles" $view_profiles "bucket_name" $bucket_name) | yaml >}}
	{{<- /* add the minio policy to the list of policies */ ->}}
	{{<- $policies = $policies | append (dict "name" $policy_name "policy" $policy) >}}
	{{<- end >}}
	{{<- end >}}
	{{<- end >}}

Some of the important templates which are called in the above section are:

• runtime/deploykf_profiles__users_id_mapping_json:
  • a JSON mapping from user -> user info (needed to get the email of the user, for the policy name)
• kubeflow_pipelines.object_store.user.minio_policy:
  • the actual per-user IAM policy template (this would need extending with the capability to take extra_statements)

You would also need to use the group -> user mapping JSON template, to get the list of users in a group:

• runtime/deploykf_profiles__groups_id_mapping_json

[Diddy42]

@thesuperzapper Thank you for the detailed answer!

I am aware of the limitations of the embedded MinIO; still, the way deployKF automatically handles the access policies is a really useful feature, and it would be even more powerful if we could extend it to buckets other than the pipelines one.

I'd be interested in contributing, so thank you for the guidelines you provided; I'll look into it!