Bucket space usage #3109

s3rj1k · 2020-03-06T19:12:45Z

I am not confident that minio/minio#6577 answers original question fully, mc admin info play does not show per bucket usage, only combined bucket usage of whole cluster.

Other solutions that were provided by minio team feel hacky, listening to bucket notifications.

Also bucket notifications as per documentation do not work for usage case when minio is used as gateway for s3, with or without cache.

The text was updated successfully, but these errors were encountered:

harshavardhana · 2020-03-06T19:19:43Z

mc admin info based bucket usage is not for gateway but for only server deployments. Gateway you are using a cloud vendor who will capture the usage for you, not us.

s3rj1k · 2020-03-06T19:36:52Z

@harshavardhana I am talking here about regular cluster (server deployments). I am not seeing any per bucket usage, only global usage.

Please stop abusing authority and dismiss issues without reading them thoroughly.

Gateway was just an example on how documentation is lacking.

harshavardhana · 2020-03-06T20:00:43Z

@harshavardhana I am talking here about regular cluster (server deployments). I am not seeing any per bucket usage, only global usage.

@s3rj1k that is available as an admin API https://github.com/minio/minio/blob/master/pkg/madmin/info-commands.go#L172

Please stop abusing authority and dismiss issues without reading them thoroughly.

The reason to close the issue is lack of information - the issue template is there for a reason, which you didn't use. It is in your best interest to provide us enough information including what you did and what you didn't to see the issue you are talking about, including server version that you are running, etc. There is nothing I see in the issue description which allows me to understand how should I help you.

We close issues every day which lack issue description, so please be mindful of our time which is limited.

Gateway was just an example on how documentation is lacking.

Gateway is not documenting this because it doesn't provide this information anyways. It is never said that it will and for some reason it is not present.

s3rj1k · 2020-03-06T20:07:18Z

2 buckets, on distributed cluster of 4 nodes, no per bucket space usage, latest minio version.

mcli  --json admin info minio
{
    "status": "success",
    "info": {
        "mode": "online",
        "deploymentID": "737a11f9-bcdb-4859-bd33-20bde21ce38c",
        "buckets": {
            "count": 1
        },
        "objects": {
            "count": 1
        },
        "usage": {
            "size": 7554990080
        },
        "services": {
            "vault": {
                "status": "disabled"
            },
            "ldap": {},
            "notifications": null
        },
        "backend": {
            "backendType": "Erasure",
            "onlineDisks": 16,
            "rrSCData": 14,
            "rrSCParity": 2,
            "standardSCData": 8,
            "standardSCParity": 8
        },
        "servers": [
            {
                "state": "ok",
                "endpoint": "s2.local:9000",
                "uptime": 276205,
                "version": "2020-02-27T00:23:05Z",
                "commitID": "5d25b10f722165fcff5ee47b241ffbcd155eebbc",
                "network": {
                    "s1.local:9000": "online",
                    "s2.local:9000": "online",
                    "s3.local:9000": "online",
                    "s4.local:9000": "online"
                },
                "disks": [
                    {
                        "path": "/storage/1",
                        "state": "ok",
                        "uuid": "9cf14857-8f8b-41f6-ba37-a050859d95a4",
                        "totalspace": 477249253376,
                        "usedspace": 12877967360
                    },
                    {
                        "path": "/storage/2",
                        "state": "ok",
                        "uuid": "c99cab5e-10de-4547-b2bc-f6a4ac3fc8c7",
                        "totalspace": 477249253376,
                        "usedspace": 12877967360
                    },
                    {
                        "path": "/storage/3",
                        "state": "ok",
                        "uuid": "27d2545e-f6d8-4423-8134-2f9ea8dcc585",
                        "totalspace": 477249253376,
                        "usedspace": 12877967360
                    },
                    {
                        "path": "/storage/4",
                        "state": "ok",
                        "uuid": "e1c88dcb-5840-46ea-9dfc-2a76cb0f3615",
                        "totalspace": 477249253376,
                        "usedspace": 12877967360
                    }
                ]
            },
            {
                "state": "ok",
                "endpoint": "s3.local:9000",
                "uptime": 276205,
                "version": "2020-02-27T00:23:05Z",
                "commitID": "5d25b10f722165fcff5ee47b241ffbcd155eebbc",
                "network": {
                    "s1.local:9000": "online",
                    "s2.local:9000": "online",
                    "s3.local:9000": "online",
                    "s4.local:9000": "online"
                },
                "disks": [
                    {
                        "path": "/storage/1",
                        "state": "ok",
                        "uuid": "61bda9e2-b8cc-4f5b-8341-cc53781367ca",
                        "totalspace": 477249253376,
                        "usedspace": 12877877248
                    },
                    {
                        "path": "/storage/2",
                        "state": "ok",
                        "uuid": "cee3dcd8-5909-4b47-8ab6-db15b0e939d6",
                        "totalspace": 477249253376,
                        "usedspace": 12877877248
                    },
                    {
                        "path": "/storage/3",
                        "state": "ok",
                        "uuid": "703ffc85-6a65-4190-a7c1-0e1d549f5a50",
                        "totalspace": 477249253376,
                        "usedspace": 12877877248
                    },
                    {
                        "path": "/storage/4",
                        "state": "ok",
                        "uuid": "e3ee3f8f-3442-43f4-92a7-8dc22861f251",
                        "totalspace": 477249253376,
                        "usedspace": 12877877248
                    }
                ]
            },
            {
                "state": "ok",
                "endpoint": "s4.local:9000",
                "uptime": 276207,
                "version": "2020-02-27T00:23:05Z",
                "commitID": "5d25b10f722165fcff5ee47b241ffbcd155eebbc",
                "network": {
                    "s1.local:9000": "online",
                    "s2.local:9000": "online",
                    "s3.local:9000": "online",
                    "s4.local:9000": "online"
                },
                "disks": [
                    {
                        "path": "/storage/1",
                        "state": "ok",
                        "uuid": "06385345-a4ed-4c9e-bedc-ad5c88c47fcf",
                        "totalspace": 477249253376,
                        "usedspace": 12877897728
                    },
                    {
                        "path": "/storage/2",
                        "state": "ok",
                        "uuid": "3b750947-5bcd-4ea7-9a31-ead160c66c80",
                        "totalspace": 477249253376,
                        "usedspace": 12877897728
                    },
                    {
                        "path": "/storage/3",
                        "state": "ok",
                        "uuid": "a0a8b044-ab89-42e5-8c9b-cf0f99b25793",
                        "totalspace": 477249253376,
                        "usedspace": 12877897728
                    },
                    {
                        "path": "/storage/4",
                        "state": "ok",
                        "uuid": "3d34b747-c624-442e-b791-4202c823fa7e",
                        "totalspace": 477249253376,
                        "usedspace": 12877897728
                    }
                ]
            },
            {
                "state": "ok",
                "endpoint": "s1.local:9000",
                "uptime": 276203,
                "version": "2020-02-27T00:23:05Z",
                "commitID": "5d25b10f722165fcff5ee47b241ffbcd155eebbc",
                "network": {
                    "s1.local:9000": "online",
                    "s2.local:9000": "online",
                    "s3.local:9000": "online",
                    "s4.local:9000": "online"
                },
                "disks": [
                    {
                        "path": "/storage/1",
                        "state": "ok",
                        "uuid": "d0ac1875-d982-46d7-a6c8-49eb44a779f7",
                        "totalspace": 477249253376,
                        "usedspace": 12877910016
                    },
                    {
                        "path": "/storage/2",
                        "state": "ok",
                        "uuid": "1144cccd-bc74-4988-90ed-bc3bbe63c47b",
                        "totalspace": 477249253376,
                        "usedspace": 12877910016
                    },
                    {
                        "path": "/storage/3",
                        "state": "ok",
                        "uuid": "79bf1237-6b3f-46bd-89e9-d56040498ebf",
                        "totalspace": 477249253376,
                        "usedspace": 12877910016
                    },
                    {
                        "path": "/storage/4",
                        "state": "ok",
                        "uuid": "4b5d01d4-9fa4-4a67-af65-ad767f86a92d",
                        "totalspace": 477249253376,
                        "usedspace": 12877910016
                    }
                ]
            }
        ]
    }
}

mcli --json ls minio
{
 "status": "success",
 "type": "folder",
 "lastModified": "2020-03-06T20:39:13.654+02:00",
 "size": 0,
 "key": "mysql-backups/",
 "etag": ""
}
{
 "status": "success",
 "type": "folder",
 "lastModified": "2020-03-03T17:22:16.439+02:00",
 "size": 0,
 "key": "s3rj1k/",
 "etag": ""
}

harshavardhana · 2020-03-06T20:10:57Z

 "buckets": {
        "count": 1
    },
    "objects": {
        "count": 1
    },
    "usage": {
        "size": 7554990080
    },

This is the bucket usage @s3rj1k - you need to use the API directly to see the bucket level histogram of various sized objects. mc admin info is just aggregating all of it.

s3rj1k · 2020-03-06T20:16:09Z

@harshavardhana You mean that no cli option for viewing this usage is available?

harshavardhana · 2020-03-06T21:05:22Z

Not yet we shall add in future in mc..

s3rj1k · 2020-03-06T21:07:03Z

@harshavardhana This should be somewhere documented as an open issue.
Thanks.

harshavardhana · 2020-03-06T21:18:17Z

Feel free to open one in our minio/mc project.

harshavardhana · 2020-03-06T21:18:40Z

Or let me move this issue to mc.

kannappanr · 2020-03-07T21:38:47Z

@s3rj1k We will be adding this command to mc soon. Until then you can use the following command

 $ mc du play/test
52B	test

This will list real time and display the value, but might take a very long time if you have millions of objects.

seqizz · 2020-05-07T15:01:53Z

Sorry for another +1'ish comment, but du is taking incredibly long on big buckets.
If I understand correctly there is already some accounting done per-bucket, waiting for the feature 🎉

stale · 2020-08-05T15:05:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

s3rj1k · 2020-08-05T17:27:59Z

Any news?

stale · 2020-11-03T18:43:38Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

s3rj1k · 2020-11-03T18:44:53Z

Still relevant

sstubbs · 2020-12-15T21:16:47Z

This would be really useful.

mrdsam · 2021-02-04T12:53:23Z

Sorry for another +1'ish comment, but du is taking incredibly long on big buckets.
If I understand correctly there is already some accounting done per-bucket, waiting for the feature

You can get the usage per bucket from the prometheus metrics https://docs.minio.io/docs/how-to-monitor-minio-using-prometheus.html

The data is not always up to date, in my setup it lags up to 24h. Nevertheless, it is a very fast way to retrieve the 'du' data.

stale · 2021-05-06T03:53:15Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

s3rj1k · 2021-05-06T10:50:52Z

Still relevant

s3rj1k · 2021-06-05T14:35:50Z

Why this is closed?

melsom · 2021-08-05T14:13:42Z

Can we reopen this? Still relevant and much needed.

geraldo-netto · 2021-08-17T12:55:40Z

Dear Friends,

I`m also interested in this feature, in my case, I would need to use it through java
How can I contribute to implement this feature?

Kind Regards,
Geraldo Netto

klauspost · 2021-08-17T13:09:30Z

We do not provide admin API through other means than madmin-go and mc.

The API is fully public in madmin, but backwards compatibility is not guaranteed for future releases. Feel free to implement it in your own language.

mirekphd · 2021-11-01T17:43:07Z

One workaround until this problem is fixed - is to maintain a local copy of the remote bucket (e.g. using backup software such as restic in parallel for remote and local copies). Then you can monitor disk usage with e.g. du -sch and df -haT on these local copies (which is performant even when local really means network storage such as NFS or Ceph). But when using minio bucket to store backup data using restic (which supports S3/minio quite transparently) the local copy is not really needed, as restic reports remote bucket disk usage automatically (printing it after each restic backup operation).

simplecoka · 2023-02-21T07:19:46Z

I think we can solve the issue with this script

package main

import (
"log"

"context"

"github.com/minio/madmin-go/v2"

)

func main() {
// API requests are secure (HTTPS) if secure=true and insecure (HTTPS) otherwise.
// NewAnonymousClient returns an anonymous MinIO Admin client object.
// Anonymous client doesn't require any credentials
madmClnt, err := madmin.New($URL,$ACK ,$ASK, false)
if err != nil {
log.Fatalln(err)
}
// Get Bucket Size
dataUsageInfo, err := madmClnt.DataUsageInfo(context.Background())
if err != nil {
log.Fatalln(err)
}
log.Println(dataUsageInfo.BucketsUsage["$bucket"].Size)

// Get Quote Size
ctx := context.Background()
quotaCfg, err := madmClnt.GetBucketQuota(ctx, "$bucket")
if err != nil {
	log.Fatalln(err)
}
log.Println(quotaCfg)

}

dminca · 2023-06-14T11:41:51Z

Isn't there any Prometheus metric exposed that provides this information? If not, probably that'd be the best implementation

s3rj1k · 2023-06-14T11:44:21Z

3 years have passed, no official solution to very basic OPS request is provided...

dminca · 2023-06-14T11:50:09Z

How about minio_bucket_usage_total_bytes ? This should give you a historical view on bucket size perhaps

seqizz · 2023-06-30T20:33:16Z

Yes, although getting updated a bit slower/incrementally, seems like minio_bucket_usage_total_bytes from mc admin prometheus metrics seems to be the easiest way of doing it currently. I got rid of the IO-heavy du solution at last.

harshavardhana closed this as completed Mar 6, 2020

harshavardhana reopened this Mar 6, 2020

harshavardhana transferred this issue from minio/minio Mar 6, 2020

kannappanr added community priority: medium labels Mar 7, 2020

kannappanr added this to the Current milestone Mar 7, 2020

stale bot added the stale label Aug 5, 2020

stale bot removed the stale label Aug 5, 2020

stale bot added the stale label Nov 3, 2020

stale bot removed the stale label Nov 3, 2020

stale bot added the stale label May 6, 2021

stale bot closed this as completed Jun 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bucket space usage #3109

Bucket space usage #3109

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020 •

edited

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020 •

edited

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

kannappanr commented Mar 7, 2020

seqizz commented May 7, 2020

stale bot commented Aug 5, 2020

s3rj1k commented Aug 5, 2020

stale bot commented Nov 3, 2020

s3rj1k commented Nov 3, 2020

sstubbs commented Dec 15, 2020

mrdsam commented Feb 4, 2021

stale bot commented May 6, 2021

s3rj1k commented May 6, 2021

s3rj1k commented Jun 5, 2021

melsom commented Aug 5, 2021

geraldo-netto commented Aug 17, 2021

klauspost commented Aug 17, 2021

mirekphd commented Nov 1, 2021 •

edited

simplecoka commented Feb 21, 2023

dminca commented Jun 14, 2023

s3rj1k commented Jun 14, 2023

dminca commented Jun 14, 2023 •

edited

seqizz commented Jun 30, 2023

Bucket space usage #3109

Bucket space usage #3109

Comments

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020 • edited

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020 • edited

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

s3rj1k commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

harshavardhana commented Mar 6, 2020

kannappanr commented Mar 7, 2020

seqizz commented May 7, 2020

stale bot commented Aug 5, 2020

s3rj1k commented Aug 5, 2020

stale bot commented Nov 3, 2020

s3rj1k commented Nov 3, 2020

sstubbs commented Dec 15, 2020

mrdsam commented Feb 4, 2021

stale bot commented May 6, 2021

s3rj1k commented May 6, 2021

s3rj1k commented Jun 5, 2021

melsom commented Aug 5, 2021

geraldo-netto commented Aug 17, 2021

klauspost commented Aug 17, 2021

mirekphd commented Nov 1, 2021 • edited

simplecoka commented Feb 21, 2023

dminca commented Jun 14, 2023

s3rj1k commented Jun 14, 2023

dminca commented Jun 14, 2023 • edited

seqizz commented Jun 30, 2023

s3rj1k commented Mar 6, 2020 •

edited

s3rj1k commented Mar 6, 2020 •

edited

mirekphd commented Nov 1, 2021 •

edited

dminca commented Jun 14, 2023 •

edited