Azure: 16% of the requests have an overhead of 500 to 1000ms, 9% have an overhead of 1 to 2s, 48% of 2 to 4s, and the remaining 27% have an overhead of 4 to 12s. The average overhead is 3097ms.

Only AWS Lambda has reasonable overhead (87ms) for concurrent requests (50), the overhead of IBM OpenWhisk (335ms) is high, the overhead for Google Cloud Functions (702ms) is even higher, but the overhead for Azure Functions (3097ms) is abysmal.

Conclusion

All providers seem to struggle with delivering constant performance, especially when concurrency comes into play. Overall AWS Lambda seems to have the best performance, mainly because of the reasonable overhead, which makes other providers pretty much unusable for applications with concurrent requests. Maybe AWS Lambda still has the competitive edge, because it started 2 years before the other services. Overhead is an important size for all kinds of requests, not only compute intensive workloads, 3 seconds of additional overhead (Looking at you, Azure) may kill the user experience.

Disclaimer and future work

This data represents only a snapshot, because it has been collected at a distinct point in time and not continuously, so it might not be as representative as it could be, when the data was collected continuously, although I think the data is capable of showing general trends.

To improve quality and validity of this data, I want to create a service that continuously collects data by requesting serverless functions, every hour and every day and provide the recent data on a dedicated website. This will cause quiet some costs for the serverless requests, so if you or your company (talk to your boss!) wants to support me in collecting this data, I plan on setting up a Patreon or OpenCollective account for that. For now if you’re interested, just leave your mail here and I will contact you as soon as the project starts.

So if you found this research, or at least some part of it, useful, please think about supporting the work.