A war story about COVID, cloud, and cost

And why serverless wins.

You’ll want to preorder two copies of my excessively colorful book, The Read Aloud Cloud — one for your desk, and one for your non-technical Uncle Mike who says: “You work in the cloud? That must be quite a commute! Ha ha ha!” and then doesn’t pass you the potato chips.

Here’s a reasonable question: if cloud really does make it just as easy to scale your infrastructure down as up, then why aren’t we reading lots of stories about companies who are successfully lowering their cloud costs right now?

One reason could simply be that this is not a story anyone is excited about telling in public. Who wants to write a Medium post called “My business is in tatters; here’s how the cloud helped me cut my losses”? (I mean sure, we all want to read that post, but I don’t know if the internet points would be worth writing it.)

Alternatively, maybe the cloud isn’t as flexible as it’s cracked up to be.

Either way, cloud adoption has only increased in quarantine. So somebody out there must be thinking hard about how to push down spend.

I asked a friend of mine about it. We’ll call her Bev. She runs the cloud division for an enterprise that’s been hit hard by COVID-19.

Bev says that cost optimization (fancy term for “lower the cloud bill NOW”) is an urgent topic where she works. And even though everybody has a billion other priorities, she’s managed to shave about $10K USD a month off her AWS bill since March.

That’s $120,000 a year. They’ve saved the equivalent of a full-time engineer. Not too shabby.

Bev runs a tight ship; it’s not like rogue devs are spinning up GPU instances to mine bitcoin on the sly. So, the natural follow-up question: under which couch cushions did you find this monthly ten grand?

Her answer: a few tweaks here and there, an increased vigilance on spend. But the majority of that cost savings came from just one action.

You ready for this?

Flipping the billing mode for DynamoDB tables, in non-prod AWS accounts only, from provisioned capacity to on-demand.

Take a minute to think about that. If it doesn’t blow your mind, take another minute. I’ll wait.

For the uninitiated, while we wait: DynamoDB tables can bill you for queries in two different ways. The older way is called provisioned capacity. You pay a monthly fee based on how many reads (RCUs) and writes (WCUs) per second you think you want to do on the table, and any overages get throttled. Just 10 RCUs and 10 WCUs on one table, a not-uncommon dev setting, can easily run you a couple hundred bucks a month depending on the size of your items. So you can see how Bev had room to save.

On-demand billing, what Bev switched to, is a newer billing mode where your DynamoDB table scales to zero. You only pay for the queries you actually make. Under sustained load, on-demand billing is approximately 7x more expensive than provisioned mode. In many dev environments, on-demand costs approach zero.

Okay, now we’re ready to walk through why your mind should be blown by this.

First, Bev gave up nothing in terms of development agility to make this switch. It’s not like she is turning off all development servers at 5 PM. If anything, on-demand billing actually makes her DynamoDB tables better in dev, because the scaling technology underneath on-demand billing is less laggy than the autoscaling you can attach to provisioned-capacity tables. So she’s paying less money to make her developers more productive.

Second, this optimization was extremely quick and cheap to implement. Changing the billing mode is a knob on the AWS console; theoretically, you could update dozens of environments by hand in a few minutes. If you are a masochist who prefers to manage everything in CloudFormation, you’d have to content yourself with a single line of YAML. So this is not a change that required a skunkworks team to burn 2-4 weeks of dev time plus ongoing maintenance (read: hundreds of thousands of dollars) to create some fancy $10K-saving automation.

And realize also that this cost savings is absolute, not relative. We’re not talking about some funny-money situation where you have to spend $100K to get $10K back. Because her workloads are now priced based on usage (not to mention fully managed), Bev’s database costs outside of production are now a rounding error. You couldn’t get this kind of optimization by, e.g., running an open-source database on your own infrastructure. (Yes, this is the ol’ “serverless” battle cry.)

Is your mind blown yet? Because we haven’t even gotten to the best part. You may be wondering, if Bev runs such a tight ship and these changes are so obviously good, why hasn’t she been running her non-prod tables in on-demand mode for years? Answer: on-demand billing mode hasn’t existed that long. It was only released in November 2018. It’s a lot newer than many of Bev’s tables.

So, in other words, this critical component of Bev’s cloud infrastructure improved under the hood without her doing anything. On-demand billing just showed up one day, at zero cost to her dev team, ready and waiting for them when COVID-19 applied the screws to her budget.

Are you getting yet how transformative the cloud is? When you bet on managed services, you accept esoteric risks like lock-in, against the reality that a large team of domain experts is going to work full-time on your behalf, at a tiny cost to you because it’s amortized over millions of users. They are going to roll out features that essentially reduce your development database spend to nothing.

And keep in mind that on-demand billing is just one of the numerous improvements (adaptive capacity, operational controls, etc) that a single AWS service team has rolled out in the last 24 months. I hope it goes without saying that this rate of feature delivery is not how business was done in your old data center, or even on your IaaS/Cloud 1.0/lift’n’shift deployments.

Sure, Bev had to have a fairly large DynamoDB footprint for this cost savings to register on her bill. But that’s why businesses continue rushing to the cloud even as they tighten their belts against the pandemic. It’s why the serverless revolution is not going away, no matter how much the name annoys us all. The advantage of shifting undifferentiated responsibility to experts at scale is simply too attractive to ignore.

So if you’re wondering, “where are all the amazing stories of lowered cloud bills?”, you’re asking the wrong question. The stories are out there, but they’re not getting much attention because superficially, they’re boring. It’s a single knob twisting $10K per month off an enterprise bill. Just another day at the cloud factory. Probably turn the knob up again tomorrow.

And of course, that’s exactly what makes the stories amazing. To me anyway.

Links and events

Gonna be lots of book talk in this space for the next little while. The latest fun news is that Simon Wardley, of all people, has somehow agreed to write the foreword to The Read Aloud Cloud; he’s both hilariously overqualified and also EXACTLY who you would want to introduce a book that explains the cloud in weird little pictures.

Pictures like this (I refuse to provide any context):

The Cloud Resume Challenge, first seen in this space, is off to a rousing start: it’s got a permanent homepage now and has been taken up by self-starting developers all over the world. This week, I recommend checking out the work of Jerry Mullis, a career changer from the front lines of health care who knocked out the challenge while also picking up four AWS certs. Somebody hire this man before he gets snapped up.

A Cloud Guru’s new webinar series has been, quite frankly, a shocking success so far. The webinars I’ve endured in the past have always felt like the worst way to consume technical information, so we’ve made sure to get a great lineup of guests, focus on digging deep into education on topics people really need to know, etc. Either it’s working or y’all are just desperate for something to do, because we keep having to raise the attendance limits in our webinar platform. You can register for the next one now - hey look, it’s on cost optimization.

Just for fun

It got a chuckle out of Werner Vogels, so I feel no shame in sharing with you “DURABLY”, a passionate piano ballad for the Simple Storage Service.