mc rm with --older-than flag removes buckets, too

[nikwen]

Expected behavior

There should be a way to delete all objects in a bucket using the --older-than flag, but not the bucket itself.
I would expect that this happens when running mc rm -r --older-than 30 --force minio/bucket/.

Actual behavior

The bucket was deleted as well. (Just for the record, it was older than 30 days.)

Steps to reproduce the behavior

1. Have a bucket older than x days.
2. Run mc rm -r --older-than x --force minio/bucket/.

Alternatively:

mc mb minio/bucket/
mc rm -r --older-than 0 --force minio/bucket/

mc version

$ mc version
Version: 2018-12-27T00:37:49Z
Release-tag: RELEASE.2018-12-27T00-37-49Z
Commit-id: 4e7eeb0725abcd36b54a37636ff89f29c3238fd8

System information

Fedora 29

[nikwen]

Workaround: mc find minio/bucket --older 30d --exec "mc rm {}"

[harshavardhana]

@nikwen this is expected behavior for the time being. do you want to preserve the buckets?

[nikwen]

@harshavardhana Thank you for your fast reply!

My usecase: I have a bucket where I regularly need to delete all objects older than x days. I do not want to delete the bucket itself and its other contents.

Improvements I would suggest:

• Add an --ignore-buckets flag to mc rm to only delete objects.
• Documentation: Adjust the flags section from the mc rm documentation, so that it says "objects and buckets":

  --older-than value            remove objects older than N days (default: 0)
  --newer-than value            remove objects newer than N days (default: 0)
  

[vadmeste]

I tend to think that we should not remove buckets so quickly, users can lose a lot of bucket configurations by mistake (bucket policy, notification, lifecycle later, etc..)

Also we need to discuss how mc should behave when versioning is enabled. Currently, it is not possible to remove a bucket with mc in that case. So --remove-bucket can be useful in this case to fix both problems, but this needs more discussion.

[nikwen]

I tend to think that we should not remove buckets so quickly, users can lose a lot of bucket configurations by mistake (bucket policy, notification, lifecycle later, etc..)

@vadmeste Totally agree!

Also feel free to change the issue title or body to whatever you need. 😉

[kannappanr]

@nikwen We have decided to add a new command

mc rb myaliaas/bucket with an optional flag --force. rb will remove a bucket. With the --force flag, it will remove it even if there are objects in the bucket.

[nikwen]

@kannappanr I like this. I assume rm will then no longer remove buckets. 😊

[nikwen]

Thank you guys. You are so awesome!

[tuscland]

Hi,

I am trying to cleanup expired objects using the following command:

mc rm --recursive --force --older-than 1d /data/upload

This also removes the upload bucket, but I would like to keep it. Is there something I am doing wrong? My objects do not have a specific prefix, I just want to scan the bucket for expired objects.

The version is RELEASE.2020-04-04T05-28-55Z.

Thanks,
Cam

[harshavardhana]

mc rm --recursive --force --older-than 1d /data/upload

This is a local folder not a bucket @tuscland

[tuscland]

Thanks @harshavardhana !
Understood, but... why should the behavior be different depending on the target?

[harshavardhana]

Yes one is behaving for a filesystem, another for an object storage both are different entities with different meaning.

[harshavardhana]

What mc does abstraction of interaction between two diametrically opposite systems, it doesn't try to generalized behavior as the concerns are different underneath for each system in question.

[tuscland]

The behavior is still misleading though because when the --fake option is specified, the list of objects that are going to be deleted is correct (it does not indicate it will delete the bucket). It's only when you actually run (not faking), that mc deletes the bucket, and even tries to delete the root folder (but fails because of underlying fs locking).

[harshavardhana]

It's not misleading it is as per what we wanted to do with mc @tuscland , you may not want it this way. But it's the way it is as per our requirements. As I stated abstraction is not generalization..

Now regarding the differentiation in what is displayed during --fake can be fixed, feel free to open a new issue for that.