I’m leaving Google Cloud at the end of this week.

Working at Google was a dream come true for me, something I’d imagined doing since I was a teenager. I met many interesting people in my 2.5 years there and got a taste of the complexity and impact of solving enormous hyperscaler-sized problems.

(I also once had to demo Vertex AI for Questlove and the Roots, and—in a semi-related incident—got a case full of video gear confiscated by the Albanian secret police. Don’t ask.)

Sometimes Google even let me make complete nonsense like this Schoolhouse Rock-style ode to S3C, which is now used as the intro to some of their cloud security workshops:

As people began to congregate again in person after the pandemic, I also began doing something completely new: I started doing talks for tech audiences that incorporated live performances of my songs. There’s something therapeutic about gathering a bunch of IT people in a room and singing about the vagaries of re-orgs or “digital transformations” or legacy code. In a weird way, it helped me rediscover the joy I had found working with technology in the first place. Other people seemed to enjoy it too.

Google was incredibly supportive of me doing this, but eventually I realized something alarming: my job was no longer the right shape for what I wanted to do. Actually, it didn’t seem likely that any job was.

I’m used to the next right thing feeling certain and obvious. That’s probably because I’m used to being young. When you’re young, there are a lot of people out there whose examples you can imitate, and relatively low risk in trying things you’re not sure will work out. You also have an unearned confidence that comes from not having failed much yet.

But even the most conventionally successful of us, young or not, may reach a place in our careers where THERE IS NO OBVIOUS NEXT STEP. The things you’ve discovered you’re good at may not exactly line up with a standard corporate career path. There may no longer be an existing, ready-made challenge that’s the right size for you to step into.

There will just be a you-shaped hole in the world—its boundaries defined by your unique connections, the extreme limit of your skills, the scope of your ambitions—and trial and error is the only way you can figure out how to fill it.

So I handed in my resignation before Christmas, walking away from what was (to me, anyway) a significant amount of money. I’m still not sure this was a smart thing to do, but it was certainly exhilarating.

What’s next for me after Google?

For the first time in my working life, I’m allowing myself a little time to discover that. I’m not closing the door on returning to full-time employment if the shape is right. And I will be doing some consulting to keep my technical skills sharp.

What I’m most certain of is that I want to make things that bring joy to people, and I have a few ideas about what that might look like that I’m excited to try. (Watch this space!)

I can tell you that while I’m exploring and experimenting, I plan to spend time on a couple of familiar things:

• Breathing some new life into the Cloud Resume Challenge (more on that soon).

• Doing more writing, speaking, and singing. Yes, I suspect my cartoons and songs are going to get even more ridiculous now that I’m self-employed. I’ve updated my speaker page, and my piano and I would be happy to regale your conference or corporate offsite.

I am grateful for what Google taught me about myself. I am happy for those who find joy and peace there. I am really excited about what I’m going to learn and build next.

If you have been wanting to collaborate with me on something, now would be a great time to get in touch!

CORBA never took over the world for a variety of reasons—not all of them technical—but the funniest excuse appears on its Wikipedia page:

See, the spec was great, it was just those silly programmers who couldn’t figure out how to apply it. The pesky fallacies of distributed computing kept getting in the way.

But software keeps needing to talk to other software over the network, and so people keep trying to design protocols and component models and service-oriented architectures to make it easier.

Microsoft tried it with DCOM (and, more recently, with Orleans1). Enterprise JavaBeans apps have been doing RPC stuff forever. The worst person you know won’t shut up about the Erlang actor model.

These solutions have made their way into lots of production systems over the years, so it would be wrong to say they “don’t work”, but nobody who’s had to support them seems very happy about it. A big reason is that distributed programming models often make a seductive promise that is impossible to fulfill: they gloss over the realities of time and space by pretending that a bunch of remote machines are the same thing as one local machine, or vice versa. (See this glorious rant about what that meant for unsuspecting DCOM developers).

In a nutshell: Distributed systems are irreducibly tricky, we’ve known this for decades, and it’s probably best that everyone stays mentally prepared for it—particularly in the cloud.

So you can see why when this week Google open-sourced a “framework for writing distributed applications” called Service Weaver, the overwhelming reaction was deja vu.

Martin Kleppmann, the patron saint of your distributed systems book club, is skeptical:

And skeptical also are hundreds of others in Kelsey Hightower’s mentions. Kelsey and the core maintainers of the project have been working overtime to listen and respond to all the feedback; here’s a Twitter Space they did yesterday that addresses a lot of common questions.

It does sound like the announcement blog and the docs could use some tweaking. In particular, I think the “write a monolith, deploy microservices” framing has gotten some people a bit off track. It seems to me that Service Weaver isn’t really a tool for creating self-contained microservices as they’re classically understood; it’s more about spreading a single service over multiple pieces of compute.

If you’re trying to figure out “why does Service Weaver need to exist”, here’s my understanding:

1. It’s Golang-first right now, and the Go ecosystem didn’t previously have a tool like this;

2. It does seem to provide better tools for dealing with some real problems people are experiencing with big Kubernetes deployments, such as upgrade versioning and testing, by consolidating code maintenance into one place while allowing atomic deployments;

3. It provides some measurable performance gains for network calls via a custom serialization mechanism;

4. It gives you some observability hooks with OpenTelemetry out of the box so you can get more insight into the distributed calls you might be making.

So I think for people who are deep into the services-on-Kubernetes world, Service Weaver has potential as a deployment tool—as long as we don’t attempt to paint it as a magic development framework that makes remote function calls behave just like local ones. That didn’t work for CORBA and it won’t work for you and me.2

On the other hand, if you are building the kind of architectures that some of us like to call “service-full” (that is, gluing cloud services together and representing the application control flow in the infrastructure diagram), then I’m more interested in something like Wing, a new programming language that interacts with cloud infrastructure the way .NET wraps Windows syscalls.

Either way, I’m cautiously encouraged that Service Weaver is a step up the abstraction ladder from “caring a lot about containers” by decoupling deployment topology from app code. And I think the widespread community interest very clearly marks it as a project to watch going forward.

In the meantime, well…

I’ve been on parental leave for a few weeks, which has been amazing for lots of reasons. It’s the first time in my career I’ve taken several consecutive weeks off work, and while it would be wrong to call it a “break”, the change of pace has been transformative. Or do I mean informative? I’m not sure. I didn’t get much sleep last night.

Anyway, at the moment my daughter is sleeping on my lap and so I have a few minutes to write up a side project I’ve been playing with.

I’m working on some fun updates to the Cloud Resume Challenge (more on that in a few weeks), and one thing I need to build is a bot to help manage the 4,000-member CRC Discord server. My requirements for the bot architecture:

• I want it to be as serverless/managed as possible, because I’m still that guy;

• I want to run it on Google Cloud, because I’m still trying to get my head around Google Cloud;

• I want to run it for free, because I want to understand the limits of the Google Cloud free tier and also because I like free things.

If you don’t know much about Discord or Discord bots, there is endless documentation out there, much of it either 1) contradictory, due to Discord’s habit of releasing vaguely similar new APIs at breakneck pace into a peculiar ecosystem of community-maintained language SDKs that are constantly winking in and out of existence like firefly lights, or 2) out-of-date, due to knock-on effects of 1). Bottom line: it’s a much more bewildering space than I expected, and I underestimated the complexity here.

So this post is going to feel a bit random, but what I’m going to try to do is talk you through my process of discovering the best way to host a Discord bot on Google Cloud and what I learned about Google Cloud’s compute options along the way.

First idea: Cloud Run

At first blush, this seems like a classic serverless use case. We want to write some Python2 that will listen to events from Discord and maybe send some scheduled messages. In the AWS world, I’d have used Lambda and API Gateway to do this. In the Google Cloud world we could use Cloud Functions, but recent updates to Cloud Functions have made clear that it’s pretty much just a programming conceit on top of Cloud Run now, so for the sake of keeping my mental models straight let’s consider Cloud Run to be Google Cloud’s default “serverless” offering.

I found Cloud Run interesting to work with, for reasons outlined in this Twitter thread:

AWS people can think of Cloud Run as a more developer-friendly, slightly more “serverless” version of Fargate; the emulated development options are nice, and it has super smooth, flexible scaling and concurrency out of the box - all the way down to zero.3

But there’s a problem. The problem is Discord’s classic way of interacting with bots, which is basically just to hold open a socket on a piece of long-running compute. (There is a newer way of talking to Discord bots that’s more event-driven - we’ll get to that in a minute; I’m sharing my learning in the order I figured it out.) That means we can’t host our bot using Cloud Run’s classic execution mode, which only spins up a CPU to process requests and scales to zero the rest of the time. Instead, we’d have to use the newer “always-on CPU allocation” feature that keeps the container running indefinitely.4 And this runs afoul of both my serverless and cost management goals. Classic Cloud Run has a great free tier; you can crunch through millions of free requests per month, just like on Lambda. Switch over to Always-On and the cheapest CPU is going to cost something like $20/month. Ugh. Let’s keep looking.5

GCE (the coward’s refuge)

For the reasons I just explained, the traditional way of hosting a Discord bot has always been just to dump it on the cheapest VM you can find. This is in fact what Google Cloud suggests, by way of a DevRel-authored tutorial blog from a year or two ago: just plop the thing on Google Compute Engine and have done with it. This has the advantage of being free: Google Cloud’s free tier lets you run one e2-micro instance at any time. That tutorial doesn’t really give a serious example, though; it suggests just SSHing into the server and executing your bot script from the command line. I’m looking for quick and cheap, but I’m not a barbarian.

It turns out that Google Cloud doesn’t have a standalone VPS product like AWS Lightsail or [insert your favorite “deploy a container to the cloud with 1 command” product here], but what they do have are container-optimized VMs. These are locked-down instances that run Docker and allow you to specify the entrypoint when you deploy the instance itself. If you want to run a single container on a single VM, for free, a container-optimized e2-micro GCE instance is the Google Cloud way to do it. And I did it.

(Pausing here to shout out the Google Cloud GitHub Actions developed by Google Cloud’s DevRel team, and specifically the GCE example workflow; they made it super easy for me to get my bot from repo to the cloud.)

So now I have a Discord bot running on Google Cloud, for free, but very much not serverless. This isn’t just a cosmetic complaint; my bot can’t scale beyond its single e2-micro instance, it’s not HA, and the long-polling architecture means all my control logic for scheduled tasks is wrapped up in my server instead of using Google Cloud Scheduler.

To boldly go where no event has gone before

Here’s where I started learning about newer Discord API modes like Interaction Events, which allow you to leave the old long-polling clients behind and just respond to HTTP POST requests like a normal web app. It would seem this would allow us to go back to Cloud Run and build a true event-driven bot, right?

Well, sort of. I’m not aware of anyone who’s built this on Cloud Run, but Gerald McAlister tried it on Lambda, and the problem he ran into was … wait for it…

(are you waiting?)

… cold starts.

Discord enforces a 3-second timeout on responses to its interaction events. Depending on your language, package size, etc, this can leave you with little time to process an I/O-bound command after a cold start. McAlister’s clever solution is a CDK construct that uses two chained Lambda functions - one to respond to the initial request and send Discord back a provisional response, the other to do the bulk of the command processing and return the final output. This is, needless to say, a hack - and I think it still might not avoid cold start timeouts in the p>99 case.

If I was building on AWS, I’d be tempted to borrow his CDK construct, but I kind of doubt this trick is worth porting over to Google Cloud. Come to think of it, most of what I need this bot to do can probably be implemented using webhooks, where the command is initiated by my app and not the Discord server. And THAT I can do - for free! - on Cloud Run.

Links and events

• The Cloud Resume Challenge has always been about helping people build something real. But beyond the OG challenge, there’s great need for more reality-informed cloud project ideas touching on security, observability, cloud automation, and more. So the Cloud Resume Challenge is now an open-source project inviting community extensions. Recent add-on projects have included a classical networking jaunt and a secure software supply chain challenge. If you have cloud knowledge and would like to propose a challenge, I hope you’ll contribute a project prompt or suggest one via issue!

• On a similar note, I keynoted Pluralsight’s Tech Skills Day, sharing my recommendations on how to build a great cloud side project.

• I chatted with The Cloudcast about building multi-cloud skills. (Spoiler: I think you should, even if you’d never consider building a multicloud app.)

Just for fun

I’ve guested on a few podcasts recently, but none more fun than this episode of The Changelog’s “Song Encoder”, where we took a tour through a bunch of my tech songs and examined the key question: why would anyone do this?

My favorite thing about that interview is that … well okay, my favorite thing is that it’s been transcribed for text, which I find much easier to consume than video. But my second favorite thing is that it gives you a peek into how you can run something in the cloud that requires simply stupid levels of global scale, while also keeping interactive latency down to a level approximating “real life”. There’s a reason that a lot of the historical innovations in computing have come out of gaming. This is a next-level problem and I recommend reading/watching the interview multiple times to pull out the nuances of what Niantic’s James Prompanya is explaining.

Hours after the interview came out, a reader of this newsletter reached out to me with the following question:

“I'd love to see a tweet thread from you decrypting [the above design], mapping the various parts to equivalent AWS services. [To be honest,] I've never really taken Google Cloud very seriously because I've never seen the architecture of a major app like this running on GCP - they always seem to be leaning on AWS more. It would be instructive, if you have the time.”

They do say adults learn primarily by analogy, and I’m right there with ya, reader - my background is all AWS, and when I see an architecture like this I immediately try to relate it to what I already know.

So why did Niantic choose to build this app on Google Cloud? Could you do it using equivalent services on AWS? What tradeoffs are they making here?

It’s hard to give too serious of an analysis based on the high level of the information available, but here are four things that jump out to me. Let’s see if we can use them to shed some new light on both clouds.

Spanner

Let’s start with Google Cloud Spanner, the boldest choice made in this design. Is there a comparable service to it on AWS?

We know that global low latency, ACID transactions, and relational semantics are important to Niantic. They’re pumping a brain-bleeding amount of traffic through this thing as well, so it has to stay fast at giant scale. What AWS database could do this? Prompanya says they evolved past Google Datastore, so DynamoDB is probably not going to work for them either.1

The closest AWS-native service to GO’s requirements would probably be Aurora, but Aurora is not going to give you the same global write-anywhere/replicate-everywhere abilities as Spanner. (You can have multi-master clusters in Aurora now, but all the instances have to be in the same region, and you can’t enable cross-region replication on them.) Think of Aurora as Postgres or MySQL on cloud steroids, but still fundamentally bound by the traditional database rules of clustering, failover, and replication. Whereas Spanner is this shapeshifting Paxos-based voodoo creature that globally distributes your data at the row level.

The tradeoff with Spanner, one that Niantic apparently was okay with, is that you’re not going to get a familiar engine to build on - it speaks SQL, but it’s a bespoke system with its own libraries and drivers. It’s worth noting that Spanner finally got Postgres compatibility in preview about two weeks ago, though.

Bottom line: I don’t think you could reproduce this piece of the app one-to-one on AWS services. If you really want a Spanner-like experience on AWS, your best bet might be to pull in a third-party service: specifically CockroachDB, which approximates Spanner’s “TrueTime” design in software to get around Google’s reliance on atomic clocks.

GKE

The Kubernetes cluster in that diagram is a bit of a black box as presented; we’re given tantalizing hints about caching and event-driven optimizations, but we don’t really know how all these services are talking to each other.

The part of my brain that used to be a Serverless Hero says “see, this is what’s so great about serverless! If you’d just built your thing out of functions, all the complexity would be forced onto the cloud infrastructure diagram, and we wouldn’t have to guess about it!”

But, of course, I’m not saying this architecture could or should have been built on functions. The near-real-time latency requirements alone might rule that out. Instead, per the interview, we get “thousands” of containers - and if you’re going to run that much Kubernetes, you probably want to let a cloud provider manage it for you.

My bias here is to say that, given the choice, GKE is a better all-around option than AWS EKS. It’s simply the most fully-featured and the most automated managed K8s service of the 3 major cloud providers, as you can see in this bake-off I helped put together for ACG a few months ago2. I’d be curious to know what Niantic’s compute bill looks like, though.

Cloud load balancing

For fairness’ sake, I should call out something here that’s been an unpleasant surprise for me when picking up Google Cloud: that even a simple static website requires you to put an always-on Cloud Load Balancer in front of the storage bucket just to get HTTPS. I don’t want my static assets to charge me 24/7 for compute costs!

Obviously what’s going on here is more complex. But still, it appears the Cloud Load Balancer / CDN / Storage part of the architecture could be replaced by Cloudfront and S3 without much conceptual tweaking.

Data and monitoring

The one place where we actually get to see a bit of message passing happening in this architecture is the Pub/Sub notification (Google Cloud’s version of AWS SNS) that triggers the decoupled data warehousing / analytics stuff. An AWS equivalent of the Bigtable → Dataflow → BigQuery pipeline in the diagram would be a streaming ETL job, something like DynamoDB → Kinesis → Glue → RedShift. Your opinions of RedShift may vary, but I suspect Niantic is quite happy using BigQuery for their data lake needs.

One last thing. Did you notice that Prompanya keeps mentioning how happy they are with using Google Cloud Monitoring as their primary pane of glass to keep an eye on all this stuff? Coming from the AWS world, that really jumped out at me. When’s the last time you heard of someone monitoring a massive-scale app in AWS using nothing but CloudWatch - not to mention evidently being quite proud of that fact?

Now, to be fair, Prompanya never says that Google Cloud Monitoring is their only window into how the app behaves, and there could be some editorial bowdlerizing going on here. But still I’m getting the spidey-sense that, like with IAM and environments, there may be some fundamental quality-of-life difference between monitoring on AWS and on Google Cloud.

Maybe in the next installment of this newsletter, I’ll have to do a deep dive into Google Cloud Monitoring and see what all the fuss is about.

Disclaimer, as always, that I’m new to Google Cloud and could be totally off on something here. If so, let me know and I’ll fix it.