C# Lambda performance vs. Node vs. Python

[bjorg]

First. Thank you for bringing C# to Lambda! I love it!

I did a quick and dirty performance test to compare how fast various lambda function can be invoked. All tests were done by sending the string "foo" as input to each respective function using the AWS console. The test was extremely simple: I just repeatedly kept clicking the Test button in the AWS Lambda Console and selected a representative log statement.

Python

REPORT RequestId: 6c2026c2-c028-11e6-aecf-116ec5921e69    Duration: 0.20 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 15 MB

Javascript/NodeJS

REPORT RequestId: 10a2ac96-c028-11e6-b5eb-978ea2c1c2bd    Duration: 0.27 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 16 MB

C#

REPORT RequestId: d9afca33-c028-11e6-99df-0f93927a56a6    Duration: 0.85 ms    Billed Duration: 100 ms     Memory Size: 256 MB    Max Memory Used: 42 MB

The functions were deployed in us-east-1 with their respective default settings. The C# function was deployed using dotnet lambda deploy-function. The Node and Python functions were deployed using the HelloWorld sample code for each respective language and changing the implementation so a simple string would be accepted and converted to uppercase.

You can find the code here: https://github.com/LambdaSharp/LambdaPerformance

Is there anything I can do to optimize invocation times on my side? Is that something you're still working on as well? Just curious on the status. Thanks!

[normj]

Performance is something we will always continue to work on with Lambda and all the supported runtimes. Every language will have their strengths and weaknesses which is why we have such fun language wars :)

This test is just measuring the language runtime startup time which is something dynamic languages like Node.js and Python have always been fast at compared to static languages like C# and Java because of the lack of type checking and lazier loading of dependencies.

[normj]

Sorry I didn't mean to close it. Feel free to continue the conversation.

[bjorg]

Thanks for keeping this open. It's not meant as a criticism, but as the foundation for an open discussion.

In your response, you mentioned that the test measure the language startup time. I'd like to get more clarity on this statement. I thought on first hit, the application is provisioned (if need be) and then launched, but on subsequent hits, it's warmed up. If it starts up every time, then there is no benefit of moving run-once code (e.g. reading config values) into the constructor of the handler class since it gets constructed on every invocation anyway. That's not how I understood though from your re:Invent presentation. Could you clarify. Thx!

[bjorg]

Just to confirm my understanding, I reread how the Lambda container works here: http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html

[mi-hol]

@bjorg @normj the documentation is quite vague in this regard :(
I've provide below feedback and hope it will be clarified

• 'How can we make it better?
  be exact on where the actual time for 'some time' can be looked-up for statement: 'AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation.'
• What are you trying to do?
  unstand for how much time its likely that reuse of a container actually happens in order to predict performance

[mrserverless]

@bjorg what you posted here isn't the Lambda cold startup time. But the warm response time.

When a C# function first starts ups from the cold, it can take up to a few hundre milliseconds (in my case it's 500ms). It once it's warmed up it will stay alive and serve requests at a faster rate similar to what you've posted (in my case it was around 0.9 to 1ms). Eventually it will die at the end of its lifetime, or autoscaling happens and more cold lambdas starts up. To get the startup time, wait 15 minutes or so then click Test.

So yes, you still put things into constructors. Because you are not getting a brand new Lambda everytime you click Test. But you will get a brand new Lambda if you click only once an hour.

In terms of optimisation, for the cold startup time you could:

1. make sure you always have requests coming in, 24x7 so that your lambdas are always warm
2. setup a periodic ping on your lambda (either cloudwatch schedules or newrelic or something), so that your lambdas are always warm.

For the warm response time there really isn't much point optimising:

1. the generally acceptable human response time is 200ms
2. less than 1 ms is actually pretty good. You would rarely get this rate outside of helloworld or toUpper.
3. If you put a API Gateway infront of your lambda and expose HTTP calls to the world. The average cached call from the web would be around 50ms.
4. AWS bills you to 100ms intervals anyway
5. 0.85ms is probably close to the raw performance of a normal peice C# code. Try running your function on your machine inside a main method and see.

Also don't confuse performance with response time. Just because Node responds in 20ms when you are manually clicking sequentially, doesn't mean it will behave the same way when there are 100k automated request flooding in per second, and increasing every second. Do some real concurrency and load tests. You may find that multi-threaded languages such as Java or C# may handle more requests per second under load. I actually saw the lambda stats some where: Python = fastest cold startup, Java = fastest warm requests per second, but can't find it now.

Anyway, this should answer your question hopefully.

[mrserverless]

found the discussion containing the benchmark https://www.quora.com/Which-language-better-suits-for-AWS-Lambda

[bjorg]

@yunspace, thanks for the detailed write-up. I'm mostly curious as to how the current .NET Core implementation marshals data from the raw internal invocation to the C# handler. Since the handler can vary in signature (both by type and number of arguments), I would assume it's invoked via reflection. So my question was basically if there was a way to get invoked without hitting the marshaling layer. For example, a Foo(Stream, ILambdaContext) signature could be a predefined handler pattern that bypasses any logic for converting the payload to a POCO instance.

Unfortunately, the invocation code isn't available for inspection, so I don't know if it could be optimized further. I would expect a warm invocation to be very close in performance to those other runtimes.

[mrserverless]

For the un-marshalling of raw input JSON into POCO, by default lambdas uses Amazon.Lambda.Serialization.Json which relies on the popular Newtownsoft Json.Net. But you can swap this default implementation out with your own serializer. See section on Handling Standard Data Types

I see what you mean. Most serializers (Json.Net included) use reflection which is slow. To prove this theory I suppose you could see if Foo(Stream, ILambdaContext) gives you better response time for warm invocations. And if so, then it's probably worthwhile to roll your own customer serializer for strings and POCOs.

[bjorg]

Actually, I wasn't talking about the data deserializer, but the method that invokes the lambda handler. Since the lambda handler can have different signatures (both in type and number of arguments), the method invoking it must rely on reflection. AFAIK, there is no way to write the lambda handler in such a way that this convenience machinery gets bypassed.

[greghroberts]

I recently did some testing on this and found the serializer really isn't making that big of an impact. In fact, removing almost all moving parts and returning static data or just empty streams, it seems like the best you can get is about 0.85ms on a warm function. I suspect the code that is invoking the C# function is likely to blame and seems only addressable by the lambda team ( @normj ).

For all the other runtimes, including Java, you can get between 0.22 - 0.30ms on a warm function. Essentially meaning the lambda invocation overhead is 3x-4x worse for C#.

While I agree that this doesn't tell the whole story as C# will likely be faster at doing real work, this framework overhead deserves to be looked at. I'm assuming some of this could be attributed to just it being new and performance hasn't been the biggest priority. Also smells like overuse of reflection without some type of caching could be to blame.

[bjorg]

I suspect as well that this is due to early code. I wish we could see what it looks like so we could assist in optimizing it. Alternatively, having a lower-level hook to experiment with would also help.

[genifycom]

I too wish we could help. I am right on the edge just waiting to put C# core to work in a myriad of ways. This is a critical defining factor between Azure and Amazon. The server less C# approach is vastly better than having to maintain multiple EC2 machines with Windows updates, performance checks, SQL updates and so on. It is almost a fulltime job keeping these environments up to date for clients.

The server less C# approach is more cost effective, much less labor intensive, automatically scalable and reusable.

The only other outstanding issue is a cost effective scalable RDC approach now :-)

[normj]

I have passed this thread onto the service team to take a look. I mostly maintain the client tooling like these libraries and the Visual Studio integration so I can't speak much about what happens on the service side. I can tell you in my limiting viewing of the service code that the reflection use is optimized but there are always other factors to look into.

@genifycom my understanding is this thread is tracking the 0.5 ms performance difference for warm starts in comparison to other runtimes. Since Lambda bills in 100 ms increments is this performance difference blocking your usage? I'm not trying to diminish the importance of getting to the bottom of this discrepancy but the big area the Lambda team is working to improve performance right now is the cold start time.

[bjorg]

To clarify, it's not stopping our adoption of C# Lambda (or LambdaSharp, as we've christened it). It's more a source of pride. :)

[normj]

Totally understand!

[SpoonOfDoom]

Not sure if this is the right place, but we are building a web backend in C# currently, and we have cold start times of several seconds, sometimes up to ten or even more. After that first hit though, everything is fine. Is this still normal for a non-trivial C# project, or is there something we can do to reduce that? We love being able to work in our familiar .NET ecosystem, but this is giving us some headaches.

I'm assuming that it has to do with the external packages we included in the project and how they're handled by Amazon during a cold start, because an empty project seems to fare much better. I was hoping someone could shed some light one this.

[jakejscott]

@normj Are there any updates about the performance issues we are seeing?

[bitshop]

@SpoonOfDoom Did you see the advice of hitting it every 10 mins or so? That's pretty standard practice in .Net forever, back in the 90s it made my hosting biz the fastest knowing that while nobody else did - But it's very common anymore and even common tools like this - https://www.iis.net/downloads/microsoft/application-initialization - that are recommended. Lambda changes the cost model but to some degree the same challenges are faced regardless of how they engineer it.

[SpoonOfDoom]

@bitshop That's what we're doing currently. We call everything once every 10 minutes, and for the most time it helps. But we still have some spikes every few hours, where we hit the cold start again - it seems that the AWS instances running .Net Lambdas have a maximum lifetime and then a new one is used regardless of the current hot/cold status? I'm not sure.
It seems like an odd decision by Amazon to not give developers the chance to fully protect against this (for a price, at least). Sure, we only hit the peak every few hours now, but if it's hit by a customer instead of our script, the customer's still going to be annoyed and perceive our app as unreliable and/or slow.

[MZhoume]

@yunspace It seems like the template lambda function will still need a over 2000ms cold startup time.
REPORT RequestId: d1e5f56c-0ea9-11e7-bb5d-bb039f76a793 Duration: 2120.69 ms Billed Duration: 2200 ms

@normj Anything did I do wrong?

Update: So I fount out that if the the function take a string input, the startup time will be ~800ms, but if I actually picked another type for it, it will become over 2000ms.

    // 800ms
    public string FunctionHandler(string input, ILambdaContext context)

    // 2000ms, Request being the other object type
    public string FunctionHandler(Request input, ILambdaContext context)

[InsidiousForce]

I have cold startup times of ~21 seconds in 128MB for two different lamba functions in C#, both receiving SNS. I get about the same time with S3 put trigger vs SNS. Warm, it's ~2 seconds.

If I up the memory to 256M I see the cod times drop to about ~10 seconds, etc.

The Lambda logs show that I use in the 40MB total memory.

From one of the functions, total runtime:

128MB cold: REPORT Duration: 21775.92 ms Billed Duration: 21800 ms Memory Size: 128 MB Max Memory Used: 35 MB

128MB warm: REPORT Duration: 1326.76 ms Billed Duration: 1400 ms Memory Size: 128 MB Max Memory Used: 37 MB

256MB cold: REPORT Duration: 11159.49 ms Billed Duration: 11200 ms Memory Size: 256 MB Max Memory Used: 39 MB

256MB warm: REPORT Duration: 792.37 ms Billed Duration: 800 ms Memory Size: 256 MB Max Memory Used: 39 MB

384MB cold: REPORT Duration: 7566.07 ms Billed Duration: 7600 ms Memory Size: 384 MB Max Memory Used: 43 MB

384MB warm: REPORT Duration: 850.59 ms Billed Duration: 900 ms Memory Size: 384 MB Max Memory Used: 47 MB

just for giggles:

1024MB cold: REPORT Duration: 3309.12 ms Billed Duration: 3400 ms Memory Size: 1024 MB Max Memory Used: 38 MB

1024MB warm: REPORT Duration: 677.57 ms Billed Duration: 700 ms Memory Size: 1024 MB Max Memory Used: 41 MB

That's a lot of overhead for cold startup.

By comparison here's a nodejs function that does about half the work (older version before the port to C#), but the same kind of work (taking an SNS, writing something to a database, storing something to S3) but this seems true across the board with other functions we have:

128MB Cold: REPORT Duration: 262.58 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 19 MB

128MB Warm: REPORT Duration: 134.79 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 19 MB

The percentage of overhead when cold seems much more reasonable.

I'm using the Visual Studio AWS tools to upload the bundle - the code is pre-compiled for the target platform before being uploaded, right? Is there something I'm missing or is this normal? The other numbers reported here are smaller but I don't know memory allocation.

[bjorg]

The cold boot time is a problem when building Slack-bots, as Slack times out after 3,000ms. Really wish there was a way to guarantee an instance is always available.

[SpoonOfDoom]

I've opened a tech support ticket about this problem. If I learn anything useful, I'll share it here.

[mrserverless]

whilst there are many work-arounds to keep the lambda warm via cloudwatch pings etc, ultimately the cold start time should be something you are comfortable with. You can optimise cold-starts but can't avoid them. Pretty sure when auto-scaling happens, the new lambdas will be scaling up from cold as well.

@bjorg to guarantee an instances is always available, probably better to consider EC2 or ECS

@InsidiousForce I'm surprised you are getting 21s cold starts. I strongly suspect it is your code rather than lambda itself. Do you have a heavy initialization block? If you run your code or unit tests locally in your VS, how long does it take? Anyway I'm only speculating, since I don't know your code. It's probably best if you raise a tech support ticket like @SpoonOfDoom

[bjorg]

@yunspace heresy! :)

[SpoonOfDoom]

Okay, after lengthy back and forth with the friendly tech support staff at Amazon, the basic takeaway from the conversation is this:
You can try to optimize your code - reduce filesize, throw out libraries that are not absolutely necessary, try to have as little static and other things that get initialized on startup as possible, and of course increase allocated memory to increase CPU power, as discussed in this thread. Keeping the Lambda alive by calling it regularly can stretch out the problem, but not eliminate it. An interval of 4 minutes seems to be the best value for most cases, but apparently the underlying behaviour is not fully deterministic, so it's not reliable rule. And even then, it doesn't fully eliminate the issue.

The bottom line, unfortunately, is that you can only get so far by doing all this, and at some point allocationg more memory stops being feasible. You will always have these cold start times, and while you might be able to reduce them, you probably won't be able to reduce them to a point where it's reasonable for a public facing API or something like that - it seems that if you can't afford to just wait every now and again, then C# Lambda is not for you, and you're better off with an EC2 instance or maybe (as we are doing now) Elastic Beanstalk, if you want easy deployment and automatic scaling.
We are now converting our Lambdas into an ASP.NET web api project, which allows us to still do comfortable deployment from Visual Studio and keep some automatic scaling options.

TL;DR: Doesn't seem like there's a reliable way around this issue. Use EBS or EC2 instead.

[normj]

Closing for lack of activity. Also this repo is mostly for supporting the client libraries and tools. A better place for this discussion is the Lambda forums where the Lambda service team monitors.

[Porubay]

Hello everyone!
I made an article were I compare Python vs PHP vs Java vs Ruby vs C#.
Can you check it and say your opinion about it? (This is not a purely technical material, but more generalizing for beginners)
https://www.cleveroad.com/blog/python-vs-other-programming-languages

[genifycom]

This article was not helpful for me.

Firstly, it was not a comparison. The title on the page says "ADVANTAGES OF USING PYTHON OVER OTHER LANGUAGES" so it is clearly NOT a comparison.

Secondly, THERE IS NO ONE LANGUAGE THAT FITS ALL REQUIREMENTS!

For example, building complex frameworks with a scripting language is incredibly difficult. Your point about Python as in "We can say that Python is a minimalistic language. It is very easy to write and read. And when it is time to think about a problem, a developer can focus on the issue, not on the language and its syntax." does not even start to help with rich models and interactions between model components.

Although we seem to have descended into thinking micro-services will answer everything, I have been in this game long enough to know that that is a very naive approach.

Problems come in MANY forms.

[SpoonOfDoom]

I agree with genifycom. In addition to that, there are some points that seem dubious if not flat out wrong. For example, saying that you can't do network-base apps (I assume that means distributed computing?) in Python seems uninformed, and stating that C# doesn't have many libraries available is a strange statement considering there are roughly 100k packages available on Nuget alone.
Also, Python doesn't have "one way to solve a problem" - there are always multiple ways to solve problems, and that usually doesn't have anything to do with the language at all.
Then there's one bit where you seem to contradict yourself, where you say in your chart that Python can't do cross-platform apps (huh? That seems wrong) and then in the text "Python is compatible with almost all modern operating systems".
There's also other smaller issues - for example, you can theoretically code in C# with Notepad and compile via command line, no IDE needed, while a proper IDE like IntelliJ also makes Python development far easier.

In addition to all that, I'm not sure that a GitHub issue with almost a year of inactivity is the proper way to advertise your blog.

[Porubay]

genifycom and SpoonOfDoom Thank you for your opinion, I will take this into consideration.

[bjorg]

Here's another opinion more favorable to .Net Core:
https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581

[niraj-bpsoftware]

Any tips on how to make c# code more efficient from .NET perspective for lambda handlers?
Some of the example questions I am trying to address:

1. WIll making lambda functions static increase reuse of Lambda context and improve performance?
2. If I make Functions class ( which has all lambda handlers) singleton class, will it improve performance?
3. If I make constant/readonly variables and share it across lambda functions, will it improve performance?

If anyone has any information, please suggest

[bjorg]

@niraj-bpsoftware why do you give it a try and post your findings here? I would be very interested to see if there is a noticeable difference. To be honest, I would be quite surprised if any of these had an impact.

1. The benefit of foregoing an instantiation, which is a one-time fixed cost over the lifetime of the function seems absolutely minimal and well beyond any measurable threshold.

2. I can't speak to this as I don't it. However, if you're handlers are different lambda functions, then they share nothing to begin with since they all run in separate processes/containers to my understanding.

3. Ditto.

[ashishpatelmi]

Thank you for sharing this useful information.

AWS Lambda has become increasingly popular nowadays. I was searching to use it in NodeJS and found this interesting blog post: Implementing AWS Lambda in NodeJS.

I hope this will help someone like me.