Cost estimation of usage-based resources (e.g. AWS S3/Lambda)

[alikhajeh1]

Status quo

We call resources such as AWS S3 or Lambda "usage-based resources" as their costs depend on usage, and for them the CLI outputs prices, e.g. "Cost depends on usage: $0.20 per 1M requests". The CLI enables you to describe usage estimates in a file called infracost-usage.yml so you can enter things like storage GB estimates and get a cost estimate (instead of just seeing the price).

As of Infracost CLI v0.9.8, we're experimenting with fetching usage file values from CloudWatch.

Over to you

What do you think about how Infracost is handling usage-based resources? Does it help your use-case? Should we consider other data sources such as the billing CSVs (e.g. AWS Cost & Usage Reports) or do those require access that engineers don't usually have?

[jgrumboe]

Hi,
I think that fetching/syncing real usage data from Cloudwatch is a cool feature. It will help a lot because in the beginning creating new resources/services it will always be some kind of guessing how much usage the resources will help. With this feature, you give devs the possibility to get more accurate estimates than the first guesses.

According to sources, I'd prefer Cloudwatch also, because billing data is not always visible to devs and in larger organizations, it's centralized through billing accounts. Although billing data would be the most accurate source including discounts.
BR, Jo

[AErmie]

I definitely like the idea of being more accurate in the cost estimate. However, some customers may not want to allow direct access to their AWS account. If this feature is developed/added (which I hope it does), please also provide an "offline" method (ie. a CSV export) as an alternative.

[alikhajeh1]

@AErmie that makes sense, currently the CLI attempts to lookup the usage when it's calculating the cost estimate for that resource and it falls back (logging a debug line) to the default behavior of using the usage file if it can't - is that the offline method you had in mind or something else?

[AErmie]

Hey @alikhajeh1, I was more thinking of an offline method to produce the Usage file. Something like, a CSV export from AWS Cost Explorer (or equivalent report), and having an InfraCost CLI command to generate the Usage file from it.

Example:

infracost --generate-usage-file --input aws_cost_explorer_csv_export.csv
infracost-usage.yml created

[richardj-bsquare]

I've just recently been doing scaling test runs for a whole day and using cost explorer to estimate pricing (with the intent of rolling the values back into the usage YAML) so this feature is quite timely for me.

Syncing usage directly from AWS would be great, however, please can we allow it to estimate those usages using a date range? That means we can do things like only run a really large scale test for just one whole day, and scale back down keeping costs far, far lower than having to do a run for a whole 30 days!

[jgrumboe]

A date/time range for the Cloudwatch query is a great idea. So CI could probably fetch the last 30 days and locally you could fetch 24h or so while developing. I hope it was that way and I understood it correctly.

[tim775]

That's cool, and makes a lot of sense! What services were you getting estimates for?

[richardj-bsquare]

Quite a relatively complex environment consisting of (but not limited to) RDS Serverless, Elasticache, SQL queues, IoT, ECS/EC2 (and all that entails), S3, Cloudwatch, etc... i.e. a great example of something that's practically impossible to estimate without historical/empirical data.

[yawboateng]

will be awesome if we could also do this in GCP 🙏

[alikhajeh1]

@yawboateng which TF resources would find most useful to see for that? We can check what we can get from https://cloud.google.com/monitoring/docs/

[yawboateng]

These would be very useful

google_bigquery_dataset
google_compute_router_nat
google_container_registry
google_storage_bucket
google_compute_global_forwarding_rule
google_compute_target_http_proxy
google_compute_target_https_proxy
google_sql_database_instance
google_container_cluster
google_container_node_pool
google_dns_record_set
google_pubsub_subscription
google_sql_database_instance
google_pubsub_topic

[alikhajeh1]

One of the community members has been hacking the atlantis diff script so the --sync-usage-file option could be run in CI, as that has creds to connect to the cloud, and it cuts down on the manual steps that need to happen on the dev machines. Their feedback:
"But longterm I think it would be good if this was automatic - out of sight out of mind type of thing where this will just work automatically to supplement the already great estimates."

[anhdle14]

I think there could be 3 possibilities here:

1. Automatic calculate everything based on the metrics provided by each cloud and check the billing directly for consumption and assumption. I am pretty sure this could be leverage to use AI to calculate the consumption.
2. Partial Automation. Generate the average, min, max load for each resources based on some key factors (users * requests * 30 days) and just report the cost based on a range and not a fixed price. This will be a one-time generation based on last month cost (*.csv file perhap) , and the output is going to be the usage-file as well.
3. Totally manual. Using the usage-file and well free to edit any factors.

[alikhajeh1]

Thanks @anhdle14! I quite like the partial automatic option, it provides historic data for users to see and it enables them to adjust the usage-file to do what-if analysis, e.g. what if my users grow by 20% next month? what if I change to the new instance types?
In general, is what-if analysis on costs important for you? or do you ship to product and adjust based on billing data?

[anhdle14]

@alikhajeh1

what-if analysis on costs important for you? or do you ship to product and adjust based on billing data?

I think is is quite important for the team to do what-if analysis based on the usage-file. However I don't think it is that much important for the product team to taking care of that, I think it mostly suited the governance and resource planning of the organization imo. Love to hear more thoughts on how other companies or teams handle this.

[jgrumboe]

In our department, it's often like starting with an infra setup which is probably a little bit oversized and doing cost-optimization a few weeks/months later. So it's still the teams responsibility but a little bit postponed 😉

[fraserc182]

This is a great feature, thanks for looking into it. However, it would be useful to know what permissions are required for this to work. I gave our user read only cloudwatch permissions but it seems like it needs ec2, s3 & dynamoDB permissions also.
Do you have a list of permissions required for this to work?

[aliscott]

@fraserc182 good idea, we'll add a list of the required permissions to the docs and message you back here when that's done (hopefully tomorrow).

[aliscott]

@fraserc182 I've updated our docs with an example IAM policy. Let me know if this makes sense/works for you.

[fraserc182]

@aliscott thanks for this! Much appreciated

[aliscott]

This is a long comment, but I want to put it out there to see what people think about the alternative approach of using the AWS Cost and Usage Reports and equivalents in other clouds.

Use cases

We're seeing two use cases for syncing usage data:

1. Cost visibility for all usage-based resources in CI

Instead of seeing “Cost depends on usage” we can show the costs based on the last 30 days of usage. So when you make a change it doesn’t just show you the price for resources like S3 and Lambda, but the actual effect the change will have on your bill.

2. What-if analysis in the CLI

Using your actual current usage as a baseline, you could see how your bill changes if your usage increases by a factor of 10, or if you switch to different infrastructure options, e.g. different S3 storage classes, you can see how this will affect your bill based on your past usage.

Problems with current CloudWatch approach

The current experimental --sync-usage-file feature allows you to generate an Infracost usage file by syncing usage data from the cloud. For each resource it calls out to different AWS APIs and CloudWatch to query for usage data.

This approach has various challenges:

1. We are unable to retrieve all usage: some usage can’t be pulled from the APIs, and in a lot of cases only a current snapshot of the usage can be pulled so we can’t retrieve an average for the past month/other date range.
2. Some of the APIs require configuration to enable the usage data to be pulled. For example to pull S3 request data a metrics filter needs to be created for CloudWatch.
3. We would need to add support for each resource individually, since they require different API lookups, and different CloudWatch filters.

Using the AWS Cost and Usage Reports instead

One idea we have is to connect to and ingest AWS Cost and Usage Reports (or equivalents for GCP and Azure) and sync the usage data from them. This will give coverage for all resources and allow us to map the usage based on the resource IDs in these files. Since the data is in a consistent format, we will be able to support all resources quickly and the usage data will match the bill as much as is possible.

Thoughts?

It would be great to get some input on this from the community. Particularly we want to know:

1. Do you already have Cost and Usage Reports enabled?
2. What barriers are there for setting this up?
3. What are we not seeing/don’t know about this approach?

[jgrumboe]

I agree that we also have setup CUR (Cost and Usage Reports) and are using it for example with the cloudcarbonfootprinttool. They query the data of CUR through Athena and have a good doc how and what to setup incl. permissions (https://www.cloudcarbonfootprint.org/docs/aws).

What I'm wondering is if it would really work, because in "default" CUR without any cost allocation tags, you can't drill down to resource level AFAIK. You get costs and usage for a service in a particular region. So, users would need to start using cost-allocation tags on resources for infracost to fetch costs and usage for a particular S3 bucket as an example.
Don't know if that scales?!

[aliscott]

@AErmie thanks 🙏. For 2 I think we could provide an IAM policy for this (similar to what we do at the moment for CloudWatch), or even some Terraform code for creating this service account. For 3, I agree we would keep using the pricing APIs for net-new resources.

I'm curious, would using the Cost and Usage Reports method allow a more accurate cost estimation (as in, estimates based on a customer's discounted rates with the Cloud Provider)?

Yes, this would be an added benefit of it.

@jgrumboe thanks for the link! AFAIK There's an option for the CURs to include resource IDs, so you don't need the cost allocation tags. I think we would need this, but the downside is that the files are massive.

[jgrumboe]

@aliscott
I see you ... you mean that option:

We haven't activated it on the current CUR (with hourly data) we use for the CCF project.

So creating a separate one for infracost would be needed anyway. I guess a daily granularity would be a good fit. And I will also activate the Athena integration as I think it would be the best way for infracost to query the data.
Let's see how big it gets (also compared to the existing CUR we have).

[aliscott]

Thanks @jgrumboe 🙏 Yeah I'd be super interested in knowing what the size of this is.

[jgrumboe]

Neglectible I would say. Maybe because of Parquet format and it's compression.

[tim775]

This is a follow up on the previous comment since we're starting to look more closely at generating usage estimate from AWS Cost and Usage Reports. We'd like to get a sense of how people have their CURs setup, so it would be great if people could reply with:

1. Do you have the "resource ids" option enabled for you CURs?
2. What time granularity is used (monthly, daily, hourly)?
3. What is the output format (CSV or Parquet)?
4. What integrations are enabled, if any (Athena, Redshift, Quicksight)?

Thanks!

[jgrumboe]

Hi,
ad 1) We hadn't resource ids enabled, but enabled it on a test CUR to see if storage goes up significantly -> that's not the case. So, we would probably add a new CUR for infracost with resource ids enabled.
ad 2) daily
ad 3) Parquet (because of Athena)
ad 4) Athena (since we are also using https://www.cloudcarbonfootprint.org/)

[alikhajeh1]

👋 Hi everyone, we've released Actual Costs into private beta where Infracost can connect to your AWS CURs and other APIs to fetch Savings Plans and RIs. Please email me ali.hosseini@infracost.io if you'd like to work with us to enable this and discuss the options to expose usage-based costs in PR comments.