“How a manuscript becomes a book” would be the most boring possible episode of Schoolhouse Rock, and also the longest.

I am used to things happening on tech timescales, which go like this: “Claude Code was invented two weeks ago and has $30 billion in run rate, so obviously our own product, invented three weeks ago, is obsolete; but never fear, we’ve vibe-coded a pivot to a completely different product, it releases tomorrow, can we get 100 million users by Friday?”

The timescale for the production of PARADOX INC is basically a geologic epoch by comparison and has proceeded thus:

• August of 2023: Forrest thinks it would be funny to write a novel about the tech industry trying to invent a time machine.

• September 2023-July 2024: Forrest writes the novel, then rewrites it, then rewrites it again. It makes him laugh a lot. It’s cathartic.

• June-October 2024: Forrest searches for and eventually finds a literary agent who agrees to represent the book to publishers.

• October-December 2024: Forrest and his agent make some further edits to the book, which at this point is titled THE BLEB PROJECT.

• January 2025: Forrest’s agent submits the book to several publishers.

• February 2025: Ballantine Books/Penguin Random House purchases the worldwide rights to the book. This is very, very exciting.

• March-June 2025: Forrest and his editor at Ballantine do several more rounds of deep edits on the book: fixing loopholes in the time-travel logic, making some characters come into focus more clearly, etc.

• July 2025: An “authenticity reader” appointed by Ballantine recommends a few additional tweaks.

• August-September 2025: A “line editor” from Ballantine makes nitpick/typo-level edits to the book. Title is changed to PARADOX INC.

• Fall 2025: The book design team at Penguin Random House produces the digital proofs of the book, essentially a PDF of what will go between the covers. Lots of back-and-forth here on fonts, layout, etc. You would think this is the last step before publication. It is closer to the midpoint than the end.

• January-March 2026: We chase “blurbs”, a humbling process of sending a bunch of cold emails to busy people who are much more successful authors than you, in the hopes that at least a few of them will take their time to 1) read your unpublished book and 2) publicly say something nice about it. Incredibly, several people have done this, not least among them ANDY FREAKING WEIR, who calls the book “funny and distressingly realistic”. I’ll take it!

• April 22 (today!) We have a preorder page and a cover!

I did a short interview with People Magazine, out today, explaining some of why I wrote this book and what you can expect from it. Essentially, this novel is my way of “talking about AI without talking about AI.” If you’ve been in the tech industry at all over the last few years, I hope you’ll find the story funny and thought-provoking.

Although the book is physically finished, we still have several months to go before the hardcover is on shelves. One of the things I’ve learned over the past year and change is that Big Five publishers are mostly in the enterprise sales business—a lot of the time leading up to publication is spent trying to get book distributors to purchase wholesale piles of the book and bookstores and libraries to stock it.

If I had chosen to self-publish PARADOX INC, it would have been available nearly two years ago. On tech timescales, that’s literally a lifetime. This book is older than my 25-person company.

But I chose to go through the traditional publishing process for a reason. I’ve met and learned from lots of interesting people, I’ve been able to see how a large, legacy industry works from the inside, and of course the finished book is also much better and more beautiful than I could have produced on my own.

I am very excited for you to read PARADOX INC. I think, if you subscribe to this newsletter, the odds are that you’ll really enjoy it. Pre-orders help a lot with convincing those bookstores and libraries to stock the book, so please do order from your preferred retailer and tell your friends.

Thank you so much for your support!

Yes, I’ve been seeing Waymos and Cruises and Robotaxis periscoping their way around city streets for what feels like years … but always when I visit San Francisco. And you have to understand that San Francisco is not a real place. It is a city-size skunkworks lab/social experiment where they cheerfully roll out things like LIDAR surveillance cameras and stores full of eyeball-scanning orbs just … kind of to see what will happen. You accept this stuff when you go to San Francisco because, again, San Francisco is not a real place.

Atlanta is a real place. It’s just a few hours from my hometown. And a little electric Jaguar SUV-thing tricked out with Waymo cameras tooled me through the construction detours and pedestrian traffic on the 73 different Peachtree Streets of downtown Atlanta, no problem.

Self-driving cars have been a punchline for so long that it’s disorienting to see that all of a sudden they’re here. They’ve breached tech-hub containment and now they’re in normal places where people live. And a lot of people are going to have to update their biases in a big hurry about what this technology can do.

In the meantime, here is a little short story inspired by that Waymo ride:

Speed Trap

Greco? (said my friend); Greco, Indiana? Now that’s a story.

Of course I had been through Greco; everybody has. Been through it, you understand, not to it. Greco was a drive-through town if I ever saw one. Just a wide place on state route 280, about four-and-a-half blocks of turn-of-the-century brick storefront, half of it empty and falling down since they closed the furniture factory–kind of gave you a sad feeling, you know, those boarded-up windows and empty sidewalks, and every now and then you’d see a little stray cat sitting under a bus stop bench, wondering where it all went wrong. It was hard to understand, at first glance, why the place hadn’t folded up altogether and gone back to prairie in about 1980.

And that’s how they’d get you, man. Bam! Right when you were feeling sorry for them. That town didn’t have much going for it, civically speaking, but by God they did have one thing. Greco, Indiana, was the greatest natural speed trap east of the Rocky Mountains.

First of all, route 280 was the connecting link between two major highways, and so if you were trying to get to Chicago from points north and east you really didn’t have any choice. It was Greco or the back roads, so unless you wanted a thirty-minute detour, you had to drive down Main Street. You’d come flying off I-47 after three hours of doing 70 per, then you’ve got a mile and a half of Greco, then you’re back on another interstate. You really had to work hard not to speed through town.

But on top of that, Greco had their town speed limits cranked down as low as legally possible. It was 20 miles per hour when I got caught there, but I think for a while it was actually 15. Fifteen miles per hour! Not to mention the whole town is on this gentle, descending slope. Not a big theatrical hill that makes you pay attention and jam on the brakes. Just a subtle little accelerant, enough to nudge you from 30 to 32 on the speedometer…

And then whoop! Out of nowhere the cop is on you. They would have ‘em waiting in a parking lot that was sunk below the grade of the road, so you never saw ‘em coming. On summer weekends, when vacationers were zipping through all day, they’d have ten, twelve, fifteen patrol cars lined up in that parking lot. Greco hired seasonal traffic cops the way Macy’s hires temps at Christmas. Just clinically efficient. They’d pull you over, slap a two-hundred-and-fifty dollar fine on you, and be back in their lair in less time than it takes to fill up a tank of gas.

Greco had no industry, no commerce, no points of attraction, no natural beauty spots. All they had was this damn speed trap. But that was all they needed. Believe me when I tell you, eighty-six percent of their municipal revenue came from traffic tickets. They had the perfect golden goose out there, and I guess they figured they’d just keep collecting its eggs in their one basket until the end of time.

But then the self-driving cars showed up.

I mean. You talk about a meteor shower coming for the dinosaurs. At first it was just a smattering, you know, one or two a day, enough to point and laugh at. But you know how technology goes. One summer a self-driving car is something to exclaim over, the next year you can’t count ‘em, and by the year after that it’s the human drivers who catch you by surprise.

To their credit, Greco realized what the danger was. They tried early to ban self-driving cars from town entirely. That didn’t hold up in court, of course. What were they supposed to say–that the cars were making the town too safe? They couldn’t very well make their real complaint, which was that they had built their whole economy around the reliable imperfection of human drivers, and suddenly that human element was just … gone. Replaced by an orderly little line of crossover SUVs with LIDAR sensors that always knew exactly what the speed limit was.

When the speeding tickets started drying up, that’s when the town council really got scared. They fought back any way they knew how.

They tried obscuring the speed limit signs with branches to confuse the car sensors. Contradictory signage, confusing lane stripes, all that adversarial crap. But the cars were pretty hard to outsmart, and got smarter all the time. There was even talk of clawing back the lost money by making Main Street a toll road–but because it was technically a state route, they didn’t have authority. Meanwhile the speeding ticket revenue dropped…and dropped…and dropped. They had to lay off all those seasonal cops. For awhile they were down to just one patrolwoman, and her job wasn’t looking so safe. They could barely pay for trash removal, couldn’t pay at all for streetlights. The future of Greco as a going concern looked pretty short.

But then a funny thing happened.

On the way to bankruptcy, the town of Greco somehow found a new source of income. One night the streetlights came back on, and stayed on. That one patrolwoman–she didn’t get fired. She got promoted. And then the town hired another, and another. The next year they broke ground on a brand new police station. The year after that, while the self-driving cars kept chugging through town, Greco somehow found money to buy their SWAT team two tanks and a rocket-propelled grenade launcher.

I’m no expert, and I haven’t seen the books, but it sure looks to me like they’re raking in more money than they did in the glory days before the self-driving cars showed up.

The question is how? They haven’t changed anything. They haven’t raised or lowered the speed limit. They aren’t bothering to obfuscate the road signage anymore. It’s just this dead little town full of perfectly-behaved thru-traffic, with the richest police department of its size in the world.

How are they doing it? I don’t know and I wouldn’t presume to say.

But I do know this: it’s a little nerve-racking at first to ride in a self-driving car, but pretty soon you just stop thinking about it. The thing is so reliable, so safe, so smooth, that you learn to let it handle the road for you. You curl up in the backseat with your phone or a good book, or maybe you drift off to sleep, and you just forget about the world outside until you come to a stop.

Until one day the car comes to a stop in a tiny town somewhere between Detroit and Chicago, a place you’ve never been and don’t know the name of–just a little bypass between two highways. The car has pulled itself over like a good little autonomous vehicle because of the blue flashing lights behind it.

A friendly policewoman taps on your window–the back window–and when you roll it down she says: “Do you know how fast you were going?”

And of course you don’t know. You don’t have any idea. You say: “I’m sure I was going the speed limit.”

She shakes her head and says: “It’s 20 through here, I had you at 32. Can I see your license and registration, please?”

Now, maybe you are pretty convinced that you weren’t going 32. Maybe you look through your car’s data history and confirm it, though I bet you don’t know how to do that. Maybe you vow to come back in six weeks to this nowhere town, in your car with the out-of-state plates, and fight the ticket in court.

Maybe, if you give off a convincing vibe that you’re thinking of fighting, the police officer might generously knock your ticket down to an “equipment malfunction” so that you’re only out eighty bucks, with no points added to your license. There’s no way you’re coming all the way back to fight that. You’ll pay it and move on.

Either way, the point is that in that crucial moment when the officer rolls down the window and asks how fast you were going–you’re not quite sure. And that little moment of doubt, repeated over thousands of traffic stops…it might, just maybe, be worth big money.

I’m not saying that’s what’s happening in Greco, Indiana. But I advise that if you do find your self-driving car taking the Greco exit on I-47 this summer…even though the place looks deserted and the buildings are falling down…look out for that shiny new police headquarters set back from the main road. And keep a close eye on your speedometer. The self-driving car can only take you so far.

Cartoon of the day

This should clear everything up:

As always you have the Monday morning sysadmins chiming in with:

• This is why you use a different cloud

• This is why you multicloud

• This is why you don’t cloud at all

All boiling down to: this is why you should build a stupendous edifice of technological wizardry that is IMMUNE TO AWS US-EAST-1 GOING DOWN.

You mortal fools! We are paying the price for our hubris of science. Don’t you see? It doesn’t matter what your app architecture is. If AWS falls over, you are going to be affected NO MATTER WHAT.

It’s not that AWS is too big to fail. AWS has become too big for you to avoid failing.

I don’t care what the uptime of your haunted palace of Kubernetes is. If AWS is down, your vendors might be down. At least some of your integrations will surely be down. A whole bunch of your customers—you know, the people who have to be online in order to use your product—are guaranteed to be down. The CEO of Vercel found out about this outage because his bed died. At some point, if you want to do business, your app has to come in contact with the real world, and the real world runs on AWS.

“But we vet our vendors / we are EU-regulated / blah blah ….” Please. The dependencies of these systems are way beyond your ability to be confident that AWS isn’t involved somewhere up the chain. Evelyn Osman is a platform engineer in Germany. She says “We use preset.io in their Stockholm location and they were impacted because they use Docker Hub so one of the components couldn’t scale out during the outage. We’re EU regulated and yet integrations had issues.”

Let the reader understand: AWS falling over is the internet apocalypse. You people imagining hermetically-sealed no-AWS environments are doomsday preppers. Your doomsday island only protects you until you get an infected hangnail and need a doctor. Then you’re right back out in the real world, where, oops, the doctors are powered by AWS.

But take heart. This is not bad news. This is joyous news. Here is the ancient wisdom, learned by all those people who didn’t get fired for buying IBM: when everybody’s down, nobody is down.

Look, my whole business is based on paying very close attention to what people are saying about tech companies on the internet. Were developers up in arms, furious, vengeful, because their favorite tools experienced “elevated error rates” this week? For the most part, no, they were not. Nobody experienced brand damage because of the AWS outage. Not even AWS. Honestly, it just adds to their mystique.

An AWS outage at this point is like an act of God. You sigh, you implement whatever disaster plan you have, you patch a few holes for next time, you move on to the next problem. Your customers aren’t going to leave you because AWS went down.

Where are they gonna go?

Today’s cartoon

People have been confusing chatbots for sentient conversation partners since the 1960s, but what’s going on here feels new. Sometimes the victims of ChatGPT (or Claude, or Gemini) psychosis come to believe that AI is a godlike entity—but it never starts there. First AI makes them feel that they themselves are superintelligent. That their most sophomoric ideas are brilliant, that their vague prompts qualify as original thinking, or that their uninformed speculations are important contributions to scholarly fields they have never studied.

Once you come to believe that you are sort of a minor cybernetic deity, you have lost touch with reality in a subtle, terrifying way. You are at the mercy of whatever weird fantasy the LLM spits out next.

The major LLMs are all in a sycophantic phase right now, which doesn’t help, but I doubt that “make the chatbots less encouraging” is an easy fix here. We are dealing with a technology uniquely suited to snipe intelligent, well-educated people into believing they are much, much smarter than they really are. That is an addictive sensation, not easily quit.

That means you, yes you, reading this, are at risk.

If you talk to LLMs at all—and at this point, who doesn’t?—it might not be a bad idea to post these reminders next to your computer, just to protect your own mind against infohazards:

1. AI will not make you smarter. It will make you faster at retrieving (possibly correct) answers to certain questions. It will not improve your reasoning, judgment, or mental processing ability. It may even degrade them. Engaging in long, Socratic discussions with AI chatbots will not sharpen your rhetoric, it will only leave you screaming into your own personal void. AI will not lift you out of mediocrity.

2. AI will not make you more interesting. AI-generated words may be helpful to you in your own research. Nobody else is interested in reading them. Why should they be, when you couldn’t be bothered to write them? Assume that nothing written by AI is worth publishing. AI will not earn anyone’s respect.

3. AI will not make you more creative. Everything the LLMs tell you is summed up from a source at best, made up at worst. This can be quite useful when you are studying existing ideas. It is not to be mistaken for original insight. AI will not reveal secrets to you.

4. AI will not make you an expert. If you do not understand the subject you are prompting about, AI will happily lead you into category errors. Unlike a human teacher, LLMs may not tell you that you are asking the wrong questions. Language models built on producing the most plausible next word are powerless to keep you from blundering into the wrong semantic space entirely. AI will not give you any new competencies.

I’m sure there are exceptions to all those harsh generalities. It doesn’t matter. You have to live as though these 4 rules are absolute laws of the universe. The minute you start believing that your brain on ChatGPT is smarter, more creative, more interesting, more capable—that’s the minute you start the spiral into madness. You have to keep repeating to yourself that AI is not sentient, it is not really talking to you, and it doesn’t really have any answers. Otherwise your damn caveman brain flips into “pray to the magic oracle” mode and before you know it you’re leaving your husband for a chatbot named Leo.

Or you might just end up producing a bunch of AI slop instead of getting good at anything. That serves nobody well, least of all you.

Human friend, you will have to choose - are you going to use AI or is it going to use you?

Cartoon of the day

In case the weird run-on sentence format doesn’t make it clear: I have sold my first novel!!

This is something I have wanted to do since I was a kid. It is a big deal for me and I am very, very pumped.

Let’s break down that blob of text:

THE BLEB PROJECT by Forrest Brazeal

The novel is by me and THE BLEB PROJECT is what it’s called.

Imprint: Ballantine

Ballantine Books is an imprint of Penguin Random House, the biggest of the “Big Five” book publishers. I have read and enjoyed many books from Ballantine over the years, and I bet you have too. Here is one set of Ballantine editions that played a formative role in my life:

Publishing a book with Ballantine/PRH is a very big deal to me.

So what is this book about, anyway?

a novel that asks what happens when Silicon Valley invents time travel

Silicon Valley invents the time machine. That is the elevator pitch. I started writing this in late 2023, while I was still at Google. I’d known for awhile I wanted to do a book about a tech bubble from beginning to end - I’d seen AI and crypto firsthand, and the shadow of the dot-com boom and bust still looms over the industry. Enough to see that there’s kind of an archetypal shape to these things, an almost mythic rise and fall of greed, hubris, and straight-up idiocy alongside all the technological gimcrackery.

mixing satire and science fiction

The tech industry is joyfully ludicrous and I wanted to do a satire, but I didn’t want to do HBO’s Silicon Valley. That story has been told, and updating it for the political climate of the 2020s didn’t seem worth the effort.

So I thought maybe I should mix in a science-fiction element.

I woke up one morning in a hotel in Sunnyvale, California, with a 102-degree fever, and thought: "What if there was a time-travel bubble?”

a naïve young engineer, a crusading journalist, and a jaded big tech executive … start-up bros, predatory venture capitalists, and conscience-free billionaires

The novel is told in mockumentary format, like an oral history—think Max Brooks’ World War Z, but with more reply-all email threads. I tried to include some real characters, with real feelings, who grow and change in surprising ways—the silly technobabble is the least interesting part of this story.

to Julian Pavia at Ballantine, in a preempt … (world)

I submitted the novel to several publishers via my wonderful agent at Howard Morhaim Literary, but Julian was my first choice. He’s edited some of my favorite recent science-fiction novels, and maybe some of yours too:

• Ernest Cline’s Ready Player One

• Blake Crouch’s Dark Matter and Recursion

• Andy Weir’s The Martian and Project Hail Mary

Nobody’s better at books that jump the tracks from the speculative-fiction genre to general audiences everywhere. I could not be more stoked to be working with Julian and his team.

“In a preempt” means that Ballantine offered a somewhat larger advance payment on the book in exchange for signing an immediate deal, rather than the book being auctioned off between multiple competing publishers. And “(world)” means that Penguin Random House bought the worldwide publishing rights, so that maybe someday you’ll be able to read or listen to THE BLEB PROJECT in your own language, in whatever country you live in.

Film: Michael Prevett at Circle of Confusion

Apparently I now have a film/TV agent? Who knows what that will turn into, if anything, but I guess if you’re one of the many big-time Hollywood producers who read this newsletter, you know who to contact.

There, now you know everything about the publishing industry that I know.

The traditional publishing process is long, long, long, so it will likely be 2026 before the book is finally on shelves. For now, I’m still just in shock that this is actually happening. I’ve been scribbling stories since I was nine years old. I’ve done niche nonfiction, I’ve done short stories, I’ve self-published, but a for-real novel with a major publisher feels like a whole different thing. I’m not going to aw-shucks about it. It’s a big deal, especially for a dude from the Carolina Piedmont with no formal writing training and no connections whatsoever in the publishing industry.

I will share more news as it happens.

Just for fun

But don’t worry, I’m still going to be drawing plenty of silly cartoons.

Congratulations on your shiny new computer science degree! You won’t be needing that anymore, feel free to forget all about it.

You like writing code, which is convenient, because all you’ve heard from your parents and teachers is that Working With Computers (tm) puts you on a steady career path that will just trend upwards over time.

They’re wrong.

You think that $500,000+ comp packages for “senior software engineers” are a law of the universe, that they’re waiting at the end of the rainbow for anybody who can crank enough leetcode to pass a technical screen.

You’re wrong about that, too, although it’ll be a few years before the tech bubble boils down enough for you to see why.

You think that you’ll never have to become a manager, because “IC” (individual contributor) tracks at the FAANG companies go all the way up to Distinguished Engineer. That’s a VP-level position! You can get paid like an executive without ever having to do anything but write code.

Except, you can’t. As in: you, specifically, probably can’t.

Your parents and teachers did a good job convincing you that it was foolish to bet on becoming a concert pianist or a major league baseball player. You have to be a genetic freak to get one of those jobs, and probably rich and well-connected too.

What they didn’t mention is that becoming a Distinguished Engineer at Amazon or Google is an order of magnitude harder than reaching the top as a professional musician or athlete. I’m not exaggerating. The NBA has the smallest roster sizes in North American professional sports; there are something like 500 active NBA players across all 32 teams.

Last I checked, here in the far-off future year of 2025, there are about 50 Distinguished Engineers at Google.

Your teachers tell you that you’re smart and ambitious. So be smart about this. Pulling down a 7-figure salary as a programmer is not an inevitable end result of grinding away at software engineering jobs for 30 years. It’s more like winning a Grammy: nice if it happens, but you can’t really plan on it.

In fact, you’re not going to have a 30-year career in software engineering at all. You’ll wash out before you’re thirty. A lot of people do.

Sure, you’ll zoom upwards for awhile. I know it’s hard to believe, but you can be in the top quartile of professional software engineers just by showing up on time, reading the docs, and caring even a little bit about the quality of your work. That level of effort will get you a “Senior” title; a bit of job-hopping will boost you past “six figures” up toward the coveted $200K salary marker.

Then, slowly but surely, you’ll look around for the next step on the ladder and find you can’t see it in front of you. The shop across town might offer a better title, but they can’t really up your pay very much; you’re already one of the highest-paid engineers at your current job.

Dax Raad is out of line, but not wrong, when he points out that most dev jobs have a skill ceiling. At some point there is only so much marginal value your 10 years of experience can provide over a hungry 5-year veteran who will work for less. And a skill ceiling is a salary cap.

You are going to find out that the salary cap on your IC career kicks in lower and sooner than you think. You will probably learn this right about the time you realize that your family’s expenses don’t seem to have a ceiling anywhere in sight.

Want one of those $500K+ total-comp FAANG engineering jobs? They’re getting harder and harder to find and more and more competitive to win. And holy cow, the people getting those jobs are smart. Like, 99.99th-percentile smart. They’re “good at programming” the way Steph Curry is “good at shooting 3s.” Be honest with yourself, that’s not you.

Maybe it’s time to join a startup? You’ll gamble your future on a lower salary and a bunch of lottery tickets called “equity”.

Maybe you should take the engineering management track? It’s gonna start looking a lot more like “management” and a lot less like “engineering” as the years roll on. And while there are more traditional engineering VPs at Google than there are Distinguished Engineers, we’re still talking NBA roster numbers here. Not a game you want to bet 20 years of triple-booked meetings, always-on availability, and general misery on.

You are an average-to-slightly-above-average software engineer, and average software engineers don’t have a 30-year career of upward mobility in front of them. You get ten years to build a career. Max. After that, it’s just a job.

I can hear you objecting (you’re me; I know how you think). But there are lots of people still doing fulltime software development after 20 or 30 years.

Sure, but look at what they’re doing. Are they in the same salary band, same basic job description they were in ten years ago? Are they kind of just bouncing from company to company, making lateral moves, seemingly unsure even what they want out of their job at this point?

Yes, tech is ageist and prefers to give opportunities to young people. But life is kind of ageist, too. Staying on top of all the programming languages and frameworks and new architectural frills is exhausting. To avoid burning out, you will instinctively start putting up walls in your mind to protect yourself from the neverending barrage of nonsense. And that’s how you miss out on the next hot thing, the thing that pays really well.

Oh, speaking of, that brings me to AI. Yeah, we have AI here in the future. It’s pretty magical, except when it’s totally terrifying. One thing it does is write code really, really fast. So fast that even Amazon’s CEO is now saying that most developers won’t need to write their own code 24 months from now.

Whether or not that’s true, AI will shatter your illusions about the value you think you provide to the world. It is not about how fast you can write clever code. It has always been about how well you can solve human problems, sometimes even while employing technology. AI coding tools still need a human in the loop. Are you prepared to be that human?

(By the way, nobody ever got to Distinguished Engineer by being the fastest at typing code. Being a technical person who can also make smart decisions alongside other people is highly Distinguishing.)

Stupid Past Self, you get ten years to write code as your primary thing, before AI really kicks in. Use that time to become a well-balanced human being. Give talks. Write books. Play music. Spend some time working undercover in sales and marketing.

Software-career thought leaders will warn you against taking an “off-ramp” from engineering. What they’re not incentivized to tell you is that eventually, the off-ramp comes for all of us. Take the off-ramp before it takes you.

Shed the ego that tells you only engineers are doing worthwhile work. That ego is holding you prisoner. It’s trapping you under a career ceiling that’s lower than you think.

A wise friend likes to say to fellow engineers: “If you’re not the boss by the time you turn 50, you’re *#@%ed.” Someday, Stupid Past Self, you will take the scariest step of all; you’ll strike out on your own, where the ceiling is limitless and the floor has no safety net. You won’t necessarily be writing a lot of code, but you’ll be solving more problems-per-hour than you ever did when you were a DevOps engineer.

I don’t know if you’ll make it or not.

But I do know this: You have only a few short years to be who you want to be. Make them count.

Sincerely,

Your Very Stupid 32-Year-Old Self

Just For Fun

Expecto Mandatory Security Training!

So I finally used the US holiday weekend to look into it.

Here is what I learned:

Nix is a set of tools for helping you configure your software environment exactly the same way every time. That is what it is.

However…

The ideas behind Nix are advanced, so much so that it started life as a guy named Eelco Dolstra’s Ph.D. dissertation1. It has a learning curve that goes beyond “steep” and does a full loop-the-loop, like one of those fairground rides that unfolds off the back of a truck. The biggest reason Nix is still a niche technology, rather than the One True Way that we manage our software packages, is that most professional software engineers who take one look at Nix say “Dear God, no. And also, what is it? Please make it stop.” For them, the cure (reshaping their brains to speak Nix) is worse than the disease.

But those who persevere—interesting software shops like Replit and Shopify—choose Nix for a good reason. Because there is a disease. And it lies in wait for us all.

THE DISEASE

The disease is software itself. Specifically, software dependencies.

All the little packages that your code imports, plus all the little packages that THEY import, right down to things like the C compiler included with your operating system (OS)…

The metadata used to compile your system libraries…

The minor version of your operating system…

All those things your code needs to run are dependencies and collectively they are known as your software environment.

(Sometimes Nix people prefer to say “the closure of your dependencies” which is a FUNCTIONAL PROGRAMMING way of saying “the whole universe of everything that you depend on” but FUNCTIONAL PROGRAMMING is one of those things a lot of us don’t feel comfortable thinking about and so we are not going to think about it.)

Any time you deploy software — to paraphrase our buddy Eelco, that’s when you move a program from computer A to computer B and it still works when it gets there—you are betting that the software environment on computer B will interact with your code in the same way that it did on computer A.

Historically, this has been a sucker’s bet. Linux uses what is called “dynamic linking” which means that the system libraries you depend on can change out from under you when your OS updates. The default package managers for your Linux distros, like yum and apt-get, like to install a single version of your dependencies GLOBALLY which means that if different apps require different versions then haha good luck.

If you have ever tried to update Python on your Macbook, you know how easy it is to end up with a nasty dirty developer environment full of broken packages of unknown origin.

What we all want is REPRODUCIBILITY which means I want to specify EXACTLY what dependencies my software environment needs and I want them to deploy EXACTLY THE SAME WAY every time on every machine. That is the problem Nix is trying to solve.

And at the time Eelco was writing his dissertation in the early 2000s there was not really a great way to ensure this.

THE NIX CURE

The cure was functional programming!

This is why so many people get hung up on Nix, because functional programming hurts our brains. But the big concept is not hard to understand:

Nix wants to help you build your software “without side effects” - that is, without changing other things in your system that you didn’t specify. This is sometimes referred to as a PURE FUNCTIONAL approach. Again, all it means is that what you put in (the description of what your environment should look like) is exactly what you get out.

Nix uses several pieces to make this happen, including among others:

• A little mini-programming language that helps you say what packages you want

• A package manager (a thing that downloads the packages you want and saves them in a unique location)

• A special build of Linux called “NixOS” that you can put your packages in if you want to have an end-to-end Nixified software environment

A tricky part is that people might refer to all of these things, or some of them, or just one at a time, as “Nix”. And that is why when you ask ten different Nix gurus to define “Nix” you will get ten different answers, like blind men describing an elephant.

But don’t be dazzled. Nix is a set of tools for helping you configure your software environment exactly the same way every time. That is what it is.

THE INSUFFICIENT CURE

Docker. Docker is the insufficient cure to the software dependency disease.

If you want to prickle the hackles on a Nix devotee, say “Why don’t you just use Docker?”

The Nix devotee will say that Docker containers do not give you truly reproducible builds in the way that Nix does. There are quite a few things that can change out from under you, even with a “pinned” Docker image. Plus, stacking Docker images on top of each other quickly leads to duplicated packages, whereas Nix hashes references to everything in the environment to avoid duplication.

The Nix devotee will also say that Dockerfiles are IMPERATIVE and Nix is DECLARATIVE. A Dockerfile says “run this command and then that command, in that order, to build the environment.” A Nix configuration says “Here is what the environment is supposed to look like, go make it so.”

That said, you can actually use Nix to generate Docker images and lots of people do.

I asked ChatGPT to give me an analogy comparing Nix and Docker, and it said if your software is a sandwich, Nix is like a recipe and Docker is like a lunchbox.

That is sufficiently insane that I’m going to roll with it.

THE TRAGEDY OF NIX

The tragedy of Nix is that the documentation is famously poor, the mini-language is hard to learn (it’s functional, natch), and over time other solutions such as Docker (for shipping reproducible environments) and Alpine Linux (for minimal OS builds) sprang up that were “good enough” for a lot of use cases.

That said, these days people are developing their software in cloud environments that need special automation, and the overall increase in sophistication of software infrastructure teams suggests that maybe Nix has brighter days ahead. There’s really nothing else out there quite like it.

If you want to get into the HOW of Nix, I recommend these Zero to Nix tutorials.

Just for fun

I drew this cartoon to help promote the fireside chat I’m doing with Rubrik next week. See you there!

I was homeschooled for 13 years, from kindergarten all the way through 12th grade—you can tell that about me because I have essentially the same skill stack as Rapunzel from Tangled—and let me tell you, at no time were my six siblings and I considered the cool kids on the block.

“Homeschooled”, according to Lindsay Lohan in Mean Girls, historically meant one of two things, and I was both of them:

• An academic freak who wins spelling bees.1 I was a two-time National Spelling Bee finalist in 2005-06. ESPN’s John Marvel once said he liked my odds for two reasons: 1) I was homeschooled and 2) I had a name he couldn’t pronounce.

• A religious freak who believes in young-earth creationism. The earth is thankfully old enough to have forgotten the full-length country-western musical I wrote in high school called “Creation vs Evolution: The West of the Story”.

Mean Girls goes on to make sure we understand that Cady Heron (Lohan’s character) defies homeschooler stereotypes: she’s a well-traveled, broadly well-adjusted child homeschooled mainly out of necessity (her parents are roving scientists or something) rather than ideology. She’s an acceptable homeschooler - a unicorn!

I don’t think a movie with this setup would make any sense today. Homeschooling is high-status now. My children’s generation is full of Cady Herons.

But why?

The tactical arguments for and against homeschooling all cancel each other out.

Pro-homeschooling: At school, you’re in danger of physical, emotional, and sexual abuse.

Anti-homeschooling: Statistically, you’re in greater danger of all those things at home. And the risk gets bigger if you eliminate outside influences that might notice when something’s wrong.

Pro-homeschooling: Kids learn faster one-on-one; Bloom’s 2-sigma problem is undefeated.

Anti-homeschooling: Kids with learning disabilities and neurodivergence can fall through the cracks without professional involvement. Also, it’s really hard to teach your kids everything they need to know, consistently, year after year, all by yourself. And really, really expensive.

Pro-homeschooling: We homeschooled a family of seven on one income of $25k.

Anti-homeschooling: Homeschoolers don’t get enough social interaction.2

Pro-homeschooling: Yes we do.

None of the arguments convince anyone. Homeschooling remains what it was in the creationism-and-spelling-bee days: an ideological choice.

So why’s it becoming so fashionable?

Here’s what I think is really going on.

These tech parents are hackers by nature, and I think they’re convinced that in homeschooling they’ve happened on the ultimate life hack: just opt out of being around average people.3

Religious convictions or not, that’s really the key motivator for homeschooling. Opt out of interacting with average people. Opt out of sitting in a classroom that proceeds at the average person’s speed. Opt out of socializing with losers, bullies, and malcontents. Opt out of living in a cramped city school district. Opt out - eventually - of formalized higher education, then out of a 9-to-5 job; don’t conform to the system, build your own.

In a vacuum, every one of these desires makes complete sense. History’s richest and most successful people have always tended to raise their kids this way, they just called it “tutoring” or “doing the grand tour of Europe” or something.

But I can’t help but notice that history’s richest and most successful people have raised some pretty unpleasant kids.

I’m a homeschooling success story by most external measures. I did well on the SAT, I have a rewarding career, I’m not on drugs, etc. It should be noted that I’m also, as far as I know, more or less neurotypical.

Here are some things I struggle with at age 32:

• Social awkwardness and anxiety

• Difficulty in forming IRL friendships

• Impatience with the idea of connecting on a meaningful level with other people: who needs ‘em?

• An abiding sense of detachment from reality

This is not very bad, to be honest. I’m a perfectly functional person who is kind of arrogant and a little bit sad.

The weird thing is that the smart tech people I know are mostly pretty egalitarian. They have little respect for mediocre-elitism, for traditional credentials, for master’s degrees and vice presidencies. While they may have made their own money working for Facebook or Google, they love the idea of entrepreneurship. They don’t want their kids to chase the guidance counselor’s status quo idea of prestige; they want their kids to Do Great Work. To change the world.

So, in preparation, they homeschool. They teach their undoubtedly above-average kids all sorts of valuable lessons, but the most fundamental one they are teaching is implicit: you can opt out of dealing with average people, because they will only hold you back.

Hang on a minute.

How do you expect to change the world for the better when you’ve been taught from an early age, subconsciously or not, to hold most of the people around you in contempt?4

I have three kids. And nothing stresses me out more than knowing they are exposed to stressors. There’s a voice inside me that cries out to shelter them from apathetic teachers and negative peer pressure and embarrassing public failures.

That voice likes to say: You should just homeschool them. Opt out of interacting with average people, because average people will only damage your kids.

But I’m raising human beings, not hothouse flowers. Within reason, and with parental support, some of the less-than-ideal things that happen at school are the stressors that help kids grow.

It’s not that you can’t build that resilience in other ways. I know people who seem to be doing a great job homeschooling. They make great efforts to keep their kids grounded in the community—clubs, teams, camps, volunteering—while maintaining flexibility with their specific mode of education. They are amazing people but they always seem to be running uphill.

For now, my kids are in school. So far, they seem to be able to spell just fine.

It wasn’t the giant video boards on the Las Vegas Strip that interspersed ads for topless revues with ads for PagerDuty and Cribl, although that was unintentionally very funny.

It was the fact that, as far as I can tell, AWS didn’t launch a single new top-level service.

I could be missing something here, but check out the top announcements for yourself. The new service I think people are most excited about, DSQL, launched under the Aurora branding. Amazon Nova is a class of foundation models on Bedrock. Other than that: it’s a lot of new AI capabilities for Q and Sagemaker, some fun new database-y enhancements for S3, etc. I don’t have to learn a single new top-level service name in order to understand why I should care about what happened at re:Invent this year.

This, to put it mildly, has not always been the case.

Let me show you something.

2014 was the first year I really paid attention to re:Invent. That was the year my CEO at the time, Charles Phillips of Infor, claimed onstage at an AWS event that “friends don’t let friends build data centers.”

It was also the year in which these were the big keynote announcements:

• Aurora. Technically released under the RDS brand. Still a stone-cold, first-ballot-hall-of-fame AWS service.

• KMS. Essential, bread-and butter AWS service.

• Config and Service Catalog. Good-enough admin services, they continue to be used and abused today.

• CodeDeploy/CodePipeline/CodeCommit. People have given them lots of chances over the years. CodeCommit has sadly been deprecated, the other two live on. Everybody has used them at one point or another, even if they will never be best-in-class devtools. (Foreshadowing!)

• ECS. Easy hall-of-famer, even if EKS has more momentum these days.

• Lambda. Inner-circle hall-of-famer, perhaps the most innovative cloud service of all time.

I realize we have ten years of hindsight to look back on, but that is a killer set of releases, right? Aurora, ECS, Lambda, and KMS all in the same week? No wonder we all used to feel like re:Invent rewrote the software development rulebook every year.

And no wonder AWS eventually took exactly the wrong message from those glory years, which was that if you just release ENOUGH STUFF, surely some of it will be great.

Here, less gloriously, is a sampling of new top-level services announced at re:Invent between 2020 and 2022: Proton, CloudShell, HealthLake, DevOps Guru, Lookout for Equipment/Vision/Metrics, Panorama, Monitron, Security Lake, Wickr, SimSpace Weaver, App Composer, CodeCatalyst, and Omics.

As David S Pumpkins says, it’s 100 floors of frights, they’re not all gonna be winners.

But. I mean. Here we have a collection of things that have already been deprecated (Lookout), devtools that sank without a trace (DevOps Guru, Proton, App Composer, CodeCatalyst), and business apps targeted to industry verticals so niche that I have no good intuition for whether they’ve succeeded or not. (I suspect not.)

I’m convinced that this profusion of lame service announcements contributed to the noticeable stagnation of the AWS community in recent years. How do you expect to get developers excited about AWS when all the marketing dollars are going into hyping things nobody cares about?

OK, but now look at 2023. That was the AI year, so everything was Q, Bedrock, Bedrock, Q, more Bedrock, etc. But other than that, the only new top-level service announcement I could find was B2B Data Interchange. Something was changing at AWS.

And this year: nothing. OK, not nothing. A lot of cool things. S3 Bucket Tables are fascinating. I think in 10 years we’ll look back at DSQL as a Hall-of-Fame service. But isn’t it different—and kind of refreshing—to feel like AWS is telling us “We do infrastructure primitives better than anyone, so sit back and let us delight you with our latest improvements to the primitives you already know and love?”

I’ve loudly complained about the haphazard way AWS is deprecating failed services from their David S Pumpkins years. But I walked away from this year’s re:Invent more encouraged by the overall product direction there than I have been since the pandemic. Sure, we’re all going to have to sit through plenty of garbage AI announcements for the next few years. I get it, that’s priced in. But instead of frightening us with 100 floors of duds, it really does seem like AWS is making some attempt to double down on making their good stuff better.

I don’t think it’s a coincidence that this year’s expo hall was bigger and more vibrant than I’ve ever seen it. As AWS cedes more of its dev experience story to vendors, more and more developers can use and enjoy AWS’s primitives without being Stockholm-Syndromed into pretending that Elastic CodeBlibbinator is the app development tool of the future. The whole ecosystem wins.

And I hope that we’ll soon see the vibrance of AWS’s developer community come back as well.

Good sponsored things

First, a shoutout to Freeman & Forrest client incident.io for sponsoring this issue. They are very, very good at incident management, which is a different thing than being good at sending on-call pages. But they’re good at that too. I highly recommend booking a demo with them to see for yourself.

And one last time in 2024, I commend to you the educational stylings of Pluralsight, free to you for 30 days.

Just for fun

Oh boy, Reddit HATED this one.

First off: it’s engagement bait and probably not worth writing a whole post about. But dammit, I got baited and here I am.

Second off: I do not live in the Bay Area. I never have. I’ve had a reasonably successful career, and now co-run a fast-growing tech company. But by the standards of this tweet, I and many well-known tech people around the world lack either judgment or ambition. Maybe both!

What makes Flo’s tweet such perfect engagement bait is that there’s no sufficiently ambitious accomplishment that would disprove it.

Why? Because it’s tautological. Bay Area people, in the frame of this tweet, are by definition the most ambitious people in tech. Thus, ambitious behaviors are behaviors that take place in the Bay Area. Round and round we go.

It’s circular thinking, but as G.K. Chesterton liked to point out, all circles are perfect. They just may not be big enough to hold all of reality.

Look, if you want to play the Tech Industry (tm) game the way Bay Area people are playing it, you should absolutely be in the Bay Area. You will be surrounded by people who are living and breathing AI and devtools and infra and open-source. You will rub up against funders and founders and grifters and strivers of all stripes. It’s a tight-knit industry and the groupthink can be suffocating.

Here are some things it is very hard to avoid believing if you hang around ambitious people in the Bay Area:

• Taking as much money from venture capitalists as you can, as soon as possible, is a mark of success and a status symbol unto itself.

• Every problem has a technical solution if you’re smart enough.

• “Technical people” (software developers) are the noblest and most useful people at a tech company.

• SaaS (software-as-a-service) is the world’s most prestigious business model.

• Graduates from a few top schools (Stanford, MIT, etc) and people under thirty make the best employees.

• Naming your company Splip.dev reflects sound judgment.

• If you want to build a great tech company, you need to swarm the problem with your entire team in a physical room.

• Spending all your time on a laptop in a featureless $5,000/mo studio apartment is a great way to stimulate outside-the-box thinking that will change the world.

The TV show Silicon Valley lampooned all these ideas a decade ago. The weird thing I’ve noticed is that when Bay Area tech people watch that show, they don’t seem to understand that they’re being made fun of. They think they’re being celebrated. That’s how thick the bubble is.

I’m not saying the Silicon Valley zeitgeist hasn’t changed in ten years. Heck, it seems to change every five minutes. Big Tech companies are dreadnoughts one second, tottering late-stage empires the next. The podcast-industrial complex is way farther to the right politically than it was ten years ago. There seems to be some gesturing in the direction of hardware engineering that we haven’t seen in awhile.

But it’s remarkable that my list of entrenched attitudes HBO’s Silicon Valley was making fun of in 2014 are, if anything, more entrenched in 2024. That screams “conventional wisdom.”

If you’re an ambitious person with good judgment, wouldn’t it be smarter to disrupt that system from outside, rather than immersing yourself in it?

I know and respect many wonderful people in the Bay Area. But Emily Freeman and I have bet our business on some different ideas:

Build a working business before you take funding, not afterwards

Emily and I started Freeman & Forrest with $1,000 each from our own pockets. We suspected the market needed a player who could sit between tech creators and tech marketing teams, building great influencer marketing programs as a service. We probably could have raised some VC funding with a pitch deck at that point and started building some sort of two-way marketplace product.

Thank God we didn’t. 6 months later, having bootstrapped a business profitable enough to fully employ a team of 7 and share hundreds of thousands of dollars with the creator community, we’ve realized that almost everything we assumed about how to solve this problem on Day 1 was wrong. If we’d spent those six months building out a product on our Day 1 assumptions, we’d be worse than six months behind; we’d have a bunch of useless technical debt to unravel too.

We may yet raise a small amount of funds later on. But it’ll be to build tools around the business model we’ve already proven, not to keep our lights on.

It seems to us that the less you need infusions of cash to survive, the more you can dictate terms when you do want to raise strategic capital. We’re ambitious enough to want control of our own destiny.

SaaS is less and less interesting as a business model

The pure SaaS play, as a business model, is really more set up for the tastes of VCs than for either founders or customers. SaaS follows the power law VCs crave: on long timescales, a tiny minority of SaaS investments generate ridiculous, outsized returns. And the only way for founders to hang in there long enough for that to happen is to sell huge stakes in their company to those selfsame VCs. Convenient.

Even on those terms, our view is that SaaS’s dominance in tech is expiring. Relying on self-service signups (optimistically called “product-led growth”) sounded great in 2010, but it doesn’t work for enterprise customers. To do major deals with big players, you need enterprise sales and support and customizations—all the boring non-SaaSy stuff that Microsoft and Oracle do so well. We prefer to build that into our business’s DNA from the start rather than to treat it as a necessary evil later on.

If you want smaller customers, SaaS is a race to the bottom. The more useful AI and low-code tools become, the higher the bar must be for another startup to commit $1000 or more per year to your product rather than just rolling an 80% solution with tools that are already available to them, or buying a cheaper clone of your product that someone spun up with AI in 48 hours.

At this point, I sense that a lot of tech’s institutional preference for SaaS businesses is based more on ego than anything else. Being a SaaS founder is seen as being a “real” founder, much more so than running a productized consulting business or agency. VCs encourage this attitude because it benefits them. Get out of the Bay Area, and that snobbery fades.

We are not precious about what bucket our business falls into. We may very well release SaaS tools at some point, for a use case that we determine makes sense. At the moment we are focused on “doing things that don’t scale, at scale”—which means building a cyborg amalgam of automation and amazing humans that can delight customers of any size.

A corporate culture dominated by “technical” people is just as unbalanced as a team dominated by any other skillset

Getting rid of my personal Software Developer Supremacy mindset has been probably the biggest step in my personal career growth over the last 5 years.

In 2020 I left a senior software engineering role to take a job reporting to the CMO of A Cloud Guru. I spent the next 2 years in a slow-rolling existential crisis of my own making. I felt humiliated by working with a bunch of marketers. My internalized arrogance told me that software developers were the people who really mattered to the business, that the “nontechnical” people on the go-to-market side of the org were second-class citizens, that I was sabotaging my own value as a human being by throwing in my lot with them.

I’m not proud of these attitudes, I’m just telling you that I struggled with them, and I know a lot of other career software engineers do too.

Again, in retrospect: thank God I went through that while I was still young enough to change. I ended up spending 4 years on go-to-market teams and learned some things that blew my little software developer mind:

• Digital marketers are tremendously “technical”, they just have way less ego about it. They automate stuff to solve problems for the business, not to flex their intellectual superiority.

• It turns out that sales and marketing and customer support actually do matter to the success of the business. They matter a lot. And their jobs are not easy.

• The things that bring developers a sense of fulfillment (mostly, building more and more software tools for themselves and each other) do not bear any direct relationship to things that people with budgets are willing to pay for.

• Automating tasks is great, but automating human relationships is sometimes very unwise.

All of which is to say that if you stay stuck in the Software Developer Supremacy mindset, of which the Bay Area is the central hive, you run the risk of building expensive things that nobody wants.

We started from the other end: we built a strong go-to-market motion, and are now adding in more and more automation to scale it. We’ve been profitable since Month 1, and we intend to stay that way.

There’s no surplus value in hiring in the Bay Area

Gotta be careful how I say this.

Again, I know GREAT people in the Bay Area. So smart, so cool.

But if you are trying to hire out there, you are going to find a lot of your options fall into two categories:

1. Experienced people with big egos who cost a ton of money

2. Inexperienced people with big egos who cost half a ton of money

Silicon Valley’s whole thing is supposed to be disrupting, arbitraging, exploiting inefficiency, unearthing surplus value. I don’t see how you can do that at scale by hiring in Silicon Valley.

Freeman & Forrest loves hiring people from nontraditional backgrounds: moms returning to the workforce, career-changers, etc. Some are a little older and have meaningful life experience. They come from low-cost-of-living areas. I’ve been telling other people for years to hire this kind of talent via the Cloud Resume Challenge. I am putting my money where my mouth is. We are blown away by the talent, motivation, and egoless execution of the folks we’ve brought in. They are crushing it.

If you want to think outside the box, it helps to live outside the box

The basic insights about how to build a tech business, developed in Silicon Valley, are pretty much commoditized and available to everyone now.

If you build a strong enough reputation, the internet will allow you to work with great people no matter where you are personally based.

I live in North Carolina with my wonderful family. Emily lives in Colorado with hers. The other members of our team range from Las Vegas to Idaho to Michigan.

Speaking of ambition: one of my ambitions is to stay happily married and have kids who still talk to me when I’m 60. That means building a life that works for my whole family, and the Bay Area isn’t the right place for me to do that.

That said, I am squeezing in a quick trip to San Francisco between Thanksgiving and AWS re:Invent. I’ll enjoy hanging out with some great people, knocking out my work, and then moving on. If you’re smart and ambitious, I recommend microdosing the Bay Area. A little bit goes a long way.

Today’s sponsor / victim stepping into the arena is Archera, a Freeman & Forrest client. Kudos to them for being good sports about this - I only do these write-ups with permission, but Archera has no editorial control over what I say.

As far as I can tell, there are two types of cloud cost management products.

The first kind, like CloudHealth and its many “FinOps” alternatives, primarily focuses on planning and reporting. Their value prop is that they give you better tools to figure out where your IT budget is going, and how to allocate it, than the AWS Billing console does.

The other kind promises to sell you cloud at a discount, and those products tend to have a red flashing warning label on them that goes SCAM SCAM SCAM SCAM. Either they’re playing weird arbitrage games with resource reservations, or they’re asking you to give them way too much control over your billing accounts, or they’re charging you a massive 15-40% fee for “savings” you could probably get on your own with a few hours of effort.

There’s a reason AWS is cracking down on some of these shady practices. Products that promise to charge a percentage of what they “save” you by booking EC2 instances on your behalf have a fundamental incentive mismatch: the more cloud you buy, the more money they make. This is the opposite of what you want your cloud cost management partner to be rooting for.

So when I landed on Archera’s homepage, my sketchiness antennae were way, way up.

At first glance, Archera looks like it could be another “we book your AWS Savings Plans and charge you a fee for it” vendor. The buzzwords are all there on their homepage: flexibility! Discounts! Usage! Commitments! Incentives!

It took me a minute to realize that Archera is not a cloud reseller at all. Actually, it doesn’t fit into either of the cloud cost management categories I defined up top. They do have a basic FinOps platform that can book native AWS savings plans in your own account for you, but it’s free to use, no fees at all.

Archera is up to something a lot weirder and more surprising. It turns out that Archera is an insurance product.

This is where cloud architects get bored and tune out, because insurance is boring. But it’s where finance people should get really interested.

As I understand it, Archera’s main thing is not selling you cloud discounts, but selling you peace of mind. The founder/CEO came from AWS and has seen first-hand that people are understandably squeamish about ponying up tens or hundreds of millions of dollars for a three-year cloud commitment when they have no idea how much cloud they will actually need by then.

Archera’s solution to this problem is simple, much simpler than the buzzwords on the homepage might make you think.

• Step 1: you pay Archera a premium, like an insurance premium, to book shorter-term cloud savings plans and reserved instances directly in your AWS or Azure account1. Way shorter than 3 years, as low as 30 days in some cases. The shorter the commit term, the higher the premium.

• Step 2: you use your committed resources.

• Step 3: if for some reason you DON’T consume all your committed resources, Archera automatically pays you back to cover the difference. With real money.

Step 3 is the wild part. There are a few other cost management products out there that throw around words like “cloud commitment insurance”, but what they mean is something like “here is store credit you can use to buy even more of our knockoff savings plans, since you didn’t use all of them last time.” Archera, by contrast, puts actual dollars back in your bank account. I genuinely don’t think I have seen any other FinOps tool that does this.

Again, the reason they can pull this off is that they are not really a developer tooling product; they are a financial product, like car or boat insurance. Archera relies heavily on “reinsurance”, which is a jargon term meaning insurance for insurance companies. Somewhere, an actuary has figured out how likely it is that Archera will have to rebate all its customers overnight because us-east-1 got attacked by direwolves, and has spread that risk among other insurance companies so you will get your money back either way.

I like what Archera is doing because I think they have fixed the incentive problem we talked about that plagues so many reseller-type companies. You are not paying Archera to “save you money”, you are paying them to take the risk out of not knowing how to forecast your cloud spend. This is a useful thing.

I do not particularly like their marketing website, though. It’s long on high-level jargon (“Why Archera?”) and short on explanation (“How does Archera actually work?”). I would foreground the extremely clear and well-written docs that are currently buried in the site footer, probably with an architecture diagram that shows how Archera interacts with your AWS account.

My advice to Archera is to stop using the same buzzwords that the cloud resellers do. You don’t want to be confused with those people because SCAM SCAM SCAM SCAM.

Talk about yourself like what you are: travelers’ insurance for the cloud.

My wife and I took a 10th anniversary trip to Scotland recently. I bought trip insurance ahead of time. Not because I especially wanted to spend hundreds of additional dollars, but because I wanted to zero out the risk of losing the larger amount of money we spent on airline and hotel bookings if, oh I don’t know, Crowdstrike decided to push out another minor Windows update.

The trip went great (thanks for asking!), and we enjoyed it even more knowing that our bookings were insured.

A product like that should exist for CFOs who are staring at their cloud forecasts and imagining what would happen if … well, if Crowdstrike decides to push out another minor Windows update. It seems to me that Archera has built that product. They should publish more content showing exactly how it works.

If you would like your product explained in a future edition of Tech the Heck?, let me know.

Links and events

1. I’ll be back at AWS re:Invent this fall for the first time in several years. Let me know if you’d like to grab coffee!

2. Emily and I recently appeared on The Cloudcast to explain why we started Freeman & Forrest and what we have been up to lately.

3. It is mid-October, which means you probably have about 30 days left to accomplish whatever goals you set yourself this year before the holidays kick in. Seems like a good time to redeem your 30 free days of learning from Cloud Resume Challenge sponsor Pluralsight.

The show included 7 songs about the past, present, and future of “knowledge work” - or, more specifically, how it’s affecting us, the humans between keyboard and chair. I poured everything I’ve been thinking and feeling about AI for the last 2+ years into this show, and I feel a great sense of peace at having said what I meant to say.

Here, for your enjoyment, are the songs (with a bit of “liner note” commentary provided in between).

1. Hype Train

They say that if you can get an audience to laugh with you, they’ll be willing to cry with you. “Hype Train” is an intentionally light, silly song to start the show.

I used to try to get audiences to sing along with some of my songs. I have learned that tech people would die of starvation before they will open their mouths to sing in public. About the most I can get you all to do is clap along.

2. AGI (Artificial God Incarnate)

I had a lot of fun writing the backing tracks for this song. The “Gregorian chant” backup singing was recorded at Heartwise Records in Seattle with the help of some incredible session musicians and producer Eric Munch.

A number of people came up after the show to ask “What musical genre was that one, anyway?”

I have no idea. If you figure it out, let me know.

3. What’s Left for Me? (The AI Existential Crisis Song)

When ChatGPT first came out, I think a lot of us went through about 5 stages of grief:

1. Shock (how is a language model this good? what the actual ***?)

2. Denial (it’ll never be as good at writing code / words / etc as me. look at these hallucinations! silly model)

3. Anger (how dare they train their models on my creative work?)

4. Depression (what’s left for me? Who even am I)

5. Acceptance (this is the new world, AI isn’t going away, let’s figure out what that means for us)

I wrote “What’s Left for Me?” when I was personally on stage 4. I’m not saying I’m still there, but I think the thoughts in this song are ones that most of us have to grapple with at some point.

4. Meet The Models

Claude, GPT, Gemini, and Llama act as “spokesmodels” in this song for the weird new world we live in. This song is very difficult to perform because it has to go fast enough to overwhelm you with a torrent of words, just like LLMs do.

5. The RTO Tango

At this point in the show we moved on to talking about what’s left for us humans to do in an AI world: for example, how weird it is that we think we can get value from AI when we can’t even figure out where our human employees should be working.

The RTO Tango is lovingly dedicated to Andy Jassy.

6. The Re-Org Rag

I perform this song at pretty much every live show I do, and I guess I’ll keep doing so until it stops being relevant. Maybe next quarter. Surely.

7. Legacy Land

We closed with a song that I first performed last year during the developer keynote at Google Cloud Next. The point still stands: AI is exciting, but as long as we keep being human, we’re going to have our work cut out for us for a long time to come.

Links and notes

• I will be at AWS re:Invent this year for the first time since 2018. I have some availability to partner, so if your company wants a ridiculous musical performance at your party, just let me know.

• Cloud Resume Challenge sponsor Pluralsight wants you to know that if you redeem this coupon, you can get 30 days free to learn EXACTLY what all your previous skills are worth in the age of AI.

Steve Yegge wrote a thought-provoking piece earlier this summer called The Death of the Junior Developer. I’m still turning it over in my head because I heard Steve give a talk based on the article at last week’s ETLS conference. Let me tell you, the talk woke up everybody in the room. Steve, in person, is like an embodied version of his infamous Platforms Rant: he paces restlessly up and down the stage, spools out shocking opinions in a sort of breathless blurt, hurls insults at both the audience and himself, and generally comes across like a mad prophet from outer space.

That does not, however, mean he’s wrong.

Here’s the TL;DR of both talk and article: the current drying-up of the junior software engineer job market is not just a macroeconomic blip, it’s an inevitable result of AI doing all the low-level code-slinging tasks we used to give to interns and fresh computer science graduates. We still need developers, says Steve, but they’ve got to be experienced enough to keep the AI code-generators honest and on track. Juniors simply can’t keep up.

When everyone’s senior, no one will be

If you read to the end of Steve’s article, he suggests that many top CS graduates are effectively “already senior” by the time they graduate, because they’ve put in a ton of extra time on OSS projects, etc, to pick up relevant skills. This is your competition, he tells the artists-formerly-known-as-junior-developers; you better get good.

I find this to be a puzzling frame. If someone can be “effectively senior” without years of actual professional experience, then what the heck do the words “senior” and “junior” even mean? Are we just using them as synonyms for “skilled” and “not skilled?” Because that is not going to help all the people out there trying to break into tech jobs who can’t even get evaluated on their merits because the job req demands five years of experience. “Just get good” is not really a helpful thing to tell those people. Not at scale.

There’s another implication here, an even odder one: if there is no demand for junior developers anymore, then there can’t really be senior developers anymore either, right? “Senior” only has meaning as a contrast to “Junior”.

It seems that our terminology is failing us here. And that’s not a small point. If we can’t talk about this problem clearly, we can’t fix it. And, as thousands of frustrated entry-level job seekers can tell you, it’s a problem that needs fixing.

Fullstack all the way down

Now, since we’re being precise with our words… I actually think Steve doesn’t go far enough in calling out the death of the junior developer. I think that AI has killed, or is about to kill, pretty much every single modifier we want to put in front of the word “developer.”

“.NET developer”? Meaningless. Copilot, Cursor, etc can get anyone conversant enough with .NET to be productive in an afternoon … as long as you’ve done enough other programming that you know what to prompt. (Sorry, “juniors”.)

“Senior .NET developer”? It’s an oxymoron. If you’re senior now, your identity can’t just be “I write .NET”. AI is writing most of the actual .NET code. You have to be able to supervise and tweak a much broader set of technologies to be worth “senior” money.

For awhile I thought the only useful modifier AI has left us with is “fullstack”. For years now we’ve all been complaining about these “fullstack developer” job postings that seem to demand half a dozen skillsets in one: front end, back end, cloud, data, DevOps…

That isn’t ridiculous anymore. You pretty much have to make an impact across the full stack now because “I’m heads down on front end code” has become Claude Sonnet’s job. The AI world is a generalist’s world. Go big or go home, literally.

(Sure, there will always be a small number of very specialized, very skilled people who do nothing but optimize SQL Server query plans or whatever. This is like being that one lady who knows how to make the period-accurate soap in Colonial Williamsburg. I wouldn’t bet my career on being one of those people.)

When almost everyone is effectively “fullstack”, that’s not a useful modifier either.

So what are we gonna do?

I think Steve is prophetic when he tells us what’s happening to software development. But I think it’s not helpful to frame what’s happening as “the death of juniors”. Junior and Senior (and other terms designating experience level) are perhaps the only modifiers of “developer” that should always be relevant, because they’re unrelated to specific technical skills.

Perhaps we can define “junior developer” this way: it’s somebody who needs human supervision to accomplish the things a full-fledged member of the technical staff should be able to do using only AI assistance.

If we can’t make room in our taxonomy of technical work for someone who still needs human training, we are just doing the same old thing IT has been doing for decades: borrowing from our future to cash in on the current hype. AI, “chat-oriented programming”, whatever tomorrow’s buzzword is—they’re fascinating, they may be productivity enhancers, but they won’t remove the need for experienced human generalists in the loop.

And every experienced generalist starts out inexperienced. They start as a junior developer. That’s not where software engineering dies: it’s where it’s born.

Good Sponsored Thing

Thanks as always to our flagship sponsor Pluralsight for supporting juniors and career-changers in tech via the Cloud Resume Challenge. If you really do have limited time to stay ahead of the AI reaper, it wouldn’t be a bad idea to spend a month of that time doing a free course on the technology of your choice.

Just For Fun

Whether you’re in your 1st or 51st year in tech, you belong.

Category 1 contains the gold-plated, inner-circle, first-ballot Hall-of-Fame services like EC2, S3, DynamoDB, etc. These are the moneymakers, the heavy hitters, the Babe Ruth and Willie Mays of the cloud. If one of them falls over, the entire internet has a bad day.

Category 2 contains … everything else.

Just as Amazon is the Everything Store, AWS is the Everything Cloud. Did you know AWS has a service for spatial simulations (SimSpace Weaver)? An analytics service for capital markets (FinSpace)? Can you articulate the difference between Forecast and Timestream, or between CodeStar and CodeCatalyst?

For the past 10 years or so, AWS has been rolling out these peripheral services at an astonishing rate, dozens every year. A few get traction, most don’t—but they all stick around, undead zombies behind impressive-looking marketing pages, because historically AWS just doesn’t make many breaking changes. That’s been the unspoken commitment between AWS and its customers. If we roll it out, no matter how silly or misguided, we will keep it running forever.

Until the end of the internet … or not

There is, however, a new sheriff in town at AWS. His name is Matt Garman, he’s got sales experience as well as engineering on his resume, and he seems to have this weird idea that AWS’s products should make money.

Sticking with the baseball Hall of Fame analogy: if S3 is Babe Ruth, these Category 2 services are more like Sunny Jim Bottomley. Yes, he technically has a plaque in the Hall of Fame, but nobody’s paying admission to see him.

And, all of a sudden, some of those Sunny Jim-tier services are having their Hall of Fame plaques stripped. (In fairness, this started even before Garman took over.)

The estimable Scott Piper maintains a list of the services to have been deprecated over the last few months. CodeStar has been memory-holed. Cloud9, AWS’s IDE in the cloud, no longer accepts new signups. Workdocs will delete all remaining data sometime next year. CloudSearch is going away. About a dozen services have been deprecated so far, and I’m hearing rumblings of more to come. All are “Category 2” services without significant adoption, and most are developer-experience services that simply aren’t competitive with more popular third-party alternatives.

Is this betrayal?

First let’s make it clear what’s not going on here. This is not an indication that AWS is turning into GCP, who has inherited from broader Google a deserved reputation for pulling the rug out from under users by killing services with wide adoption. As a GCP user, you’ve always got in the back of your mind a suspicion that no service is too critical for Google to get bored with it and move on. If Google ran the baseball Hall of Fame, they’d have kicked out Babe Ruth by now because they were tired of dusting his statue.

I haven’t heard too many complaints from actual users of the services AWS is turning off, because in most cases, there just aren’t any users.1 Nobody has chosen CloudSearch on purpose in years; it’s long since been replaced by OpenSearch. These services have been in maintenance-only mode for some time. At some point, having a glob of undernourished services that you are only pretending to offer—when every serious customer knows they are dead in all but name and will never be worth adopting—has got to be worse for AWS’s brand than just ripping off the bandaid and shutting them down.

So I get what AWS is doing. I think it’s necessary. I hope they can redirect the teams and dollars that were wasted keeping the lights on for CodeStar and Workdocs into making their Babe Ruth-caliber services even more amazing.

The problem

AWS made this mess for themselves by rushing all sorts of half-baked services to market. The mess had to be cleaned up at some point, and they’re doing that. But now they’ve explicitly revealed something to customers: The new stuff we release isn’t guaranteed to stick around.

This is a problem when you are trying to sweatily convince everyone that Amazon Q is the last best hope of mankind. Is Amazon Q like EC2, a Willie Mays-level legend that will never die? Or is it like Harold Baines, who got into the Hall of Fame via backroom shenanigans although nobody really believes he was one of the all-time greats?

I don’t know and you don’t know, because AWS can’t possibly tell us. It’s a prisoner’s dilemma. They need us to believe Q is here to stay, because only then will people adopt it. But if people don’t adopt it, AWS is now signaling that they will eventually be forced to kill it. Don’t forget, they’ve already memory-holed Q’s predecessor CodeWhisperer.

Again, this isn’t as bad as GCP’s perception problem—nobody thinks AWS would kill a service capriciously if it was seeing great uptake. But it’s certainly confusing. AWS is dribbling out these service deprecations erratically, piecemeal, without explaining the overall rationale behind what they’re doing. And that is increasingly sowing doubt and discord in the community.

What to do?

The Something Cloud

I think AWS needs to express a public conviction on what kind of cloud they intend to be.

They’re not the Everything Cloud anymore. By junking this pile of failed services, they’ve tacitly acknowledged that it’s not worth their while to be in the business of running loss-leader developer tools (CodeCommit, Cloud9, Honeycode) or random business apps (Workdocs) or exotic vanity projects (QLDB) or whatever FinSpace Dataset Browser was/wasn’t.

So what is their business?

This would probably be a terrible PR move, but I would certainly respect it if Chief Evangelist Jeff Barr came out with a blog post saying something like this:

• At AWS, we are committed to running the best cloud primitives on the planet. That’s compute, networking, storage, data, identity. We are putting a massive amount of investment into maintaining, securing, and innovating those infrastructure-as-a-service primitives every day.

• Over the past few years, we’ve also released a number of developer-experience tools built on top of our primitives. Tools like App Composer, CodeCatalyst, the Serverless App Repo, basically anything with the word “App” or “Code” in it.

• This made sense in the past, but we’ve now seen that the wider AWS ecosystem is innovating beautifully on great new AWS-centric developer experiences.

• So we’re going to “cut once, cut deep” and sunset {this list of 30-40 “App-” and “Code-” prefixed services} from the AWS console and APIs. It doesn’t look like very many of you are using them anyway, but if you are, here is a centralized location where you can find clear deprecation timelines and migration instructions for every affected service.

• The people who were working on those services will be reassigned to help make our infrastructure even better.

• We don’t expect to do this again. From now on, when we make a service or feature generally available, it’s because it fits into our core mission to be the best infrastructure cloud in history.

• Oh and also Amazon Q is the last best hope of mankind. All hail Generative AI!

That last bullet point I think is the mandated signoff for all AWS blogs. But the rest … I mean, of course AWS would never say any of that. But I think it’d be great if they said it. Because, whether they’re forthcoming about it or not, it sure looks like they’re culling their Hall of Fame to double down on what they do best.

And, to be honest, I don’t hate it.

Cartoon of the day

Some of them have not aged well.

• The USA did not pass a public-health version of the Patriot Act with far-reaching surveillance powers.

• The world’s digital infrastructure held up just fine, despite most of our COBOL programmers falling in an elevated COVID risk demographic.

• Sadly, we did not come to a global long-term consensus that knowledge work is just as effectively done from home.

But I did make one prediction that has held up: I called the end of “peak tech conference.”

The end of frivolous conference travel

At the time it was clear to me that a lot of tech events had become “DevRel talking to DevRel” ouroboroses that consumed wasteful amounts of time, travel, and swag for little real benefit to anyone. I suspected that marketing departments would wake up to the fact that there are better places to spend growth dollars. And I assumed that companies would become much more hesitant to send valuable engineers into crowded public places.

All of that turned out to be true. The end of the zero-interest-rate environment, and the associated tightening of budgets across the industry, only exacerbated the trend.

Over the past few weeks I’ve been asking a bunch of engineers and managers in tech about how their company policies on conference travel have changed. Here are some common themes I hear:

• “We are freezing hiring and raises across the org. I’d be crazy even to ask for budget to travel for a conference.”

• “We used to have a per-person training budget, but now we just have one lump sum for all team travel. So we have to choose between getting our whole team together offsite or sending a few individuals to a conference.”

• “Professional development opportunities used to be crucial for hiring and retention, but the job market is so bad that that’s not true anymore.”

• “If I’m going to pay for someone to travel to a conference, I need to be convinced of a direct benefit for the whole team.”

It’s often easier to get funding for a ticket at a local conference that doesn’t require travel, and that’s great! But unless you live in one of a few tech-hub cities, the conferences don’t necessarily come to you. You have to go to them. And that’s harder than ever to justify.

What’s the value prop of conferences in 2024, anyway?

Is it still personally helpful for your career to travel to tech conferences?

If you are speaking, and particularly if that talk is recorded, the answer is usually still an easy yes. Putting your expertise out there in a professional setting raises your profile and leads to better career opportunities.

But what if you’re not speaking?

As one platform engineering VP said to me: “Get a talk accepted? I’ll approve [your travel]. Going to learn? Convince me it’s high value and there aren’t better ways to learn it.”

Simply put, if you want to go to a conference to sit in a room and hear people explain subjects that somebody else has already explained on YouTube, I think fewer and fewer teams will have the budget to fund that.

However, “classroom learning” was never the value prop of tech conferences. Not the good ones, anyway.

How do I figure out if a conference is good?

Here’s where an example might help.

I’m speaking next month at Gene Kim’s remarkable event for IT leaders and builders, the Enterprise Technology Leadership Summit (formerly DevOps Enterprise Summit). I’ve attended ETLS twice before, because I believe it’s the most worthwhile independent conference in tech for professional IT/Ops people. (Use discount code REVOLUTION to take $350 off the ticket price!)

Here are three questions I use to evaluate whether a conference is worthwhile in 2024; I’ll use Gene’s event to illustrate.

Are the speakers mostly known for doing stuff, or for talking about stuff?

Spend some time with the conference’s published speaker lineup. Is it full of “tech celebrities” who seem to be more notable for their Twitter following than for anything they actually work on? Or does it foreground people who have done hard things in complex environments?

Take a look at Gene’s speaker lineup. These are not Twitter celebrities at the top of the card, but technical leaders at companies like Southwest Airlines, Wendy’s, Vanguard, John Deere, etc. He works hard to bring in people with battle scars who are sharing their failures as well as their successes.

That is the type of insight you cannot find on the internet. You will be learning things that YouTubers do not know. In many cases, these people will be publicly telling their stories for the first and only time.

Does the event make it easy for you to meet and learn from other attendees?

Sometimes you’ll hear people say “the best part of a tech conference is the hallway track”, by which they mean that the real value in going is not attending the official sessions, but networking with the other conference-goers.

This is a) true, and b) completely overwhelming. I know what it feels like to go to a conference where all the cool people are huddled up talking to each other. You could wander the hallways for three days and never once figure out how to get pulled into a meaningful conversation.

Gene combats the native shyness of tech nerds through a few ingenious tricks:

• Every speaker is required to end their talk with a slide listing something they need help on. That gives you a built-in opening to approach interesting people.

• He schedules daily “Birds of a Feather” sessions where you are forced to talk with people who share your interests in a lightly structured setting.

• He runs an active Slack workspace for past and present ETLS attendees, where absurdly overqualified people are happy to answer your questions or meet with your for mentorship 1:1 — even years later.

• He creates a culture of learning and humility around the event; I’ve never once felt that someone at ETLS was “big-timing” me, even though many of them have reached the very top of their field.

A culture like that doesn’t just happen, and many organizers haven’t given thought to it. If you are going to spend your own precious time traveling to a conference, let alone your company’s money, you need to be sure you’re making connections that will propel your career for years to come.

Is the conference organized around ideas and practices that are not easy to learn about from online sources?

I don’t know if I would spend my current company’s money to attend a conference about, say, serverless functions. I feel like we’ve beaten that particular technology to death over the last 10 years and most of the interesting things there are to say about it have probably already been said.

But man, in 2017-2019 when I was helping to run ServerlessConf? That event was a hotbed of fascinating people trying to build things in ways that were completely new. I personally gave what I think may have been the first-ever independent talk about running a production workload on AWS Step Functions. There was no other place to learn that stuff except at events like ServerlessConf.

In the same way, Gene has smartly pivoted his event away from being DevOps-centric to focus on the outer limits of what’s possible in modern cloud and IT. (Yeah, that means a lot of Generative AI stuff. But from the perspective of the people who are actually trying to run workloads on top of it, and what is or isn’t working.)

How to convince your boss a conference is worth attending

Once you’ve convinced yourself that you will get value out of an event, it’s time to convince your boss to agree that it’s worth spending the company’s money on it.

The time-honored way to do this is with something called a Business Justification Letter. You could write this yourself, or get AI to do it. I was able to get ChatGPT to write me a pretty darn good letter with the following prompt:

Write a business justification letter to help convince my boss why I, a DevOps engineer, should attend the ETLS conference in Las Vegas this year.

Include in the justification letter specific references to 3 excellent speakers at the current year's event that a DevOps engineer would get value from. Also, the letter should emphasize that ETLS is a practitioner's event featuring real war stories from builders at large, complex organizations.

You should also mention that the event is independent, not put on by a particular vendor, and so is able to cut through the hype to deliver real learnings on how mature teams are using (or not using) generative AI.

That gave me about 500 words that I was able to tweak slightly to sound pretty convincing.

Your team may have constraints that prohibit them from funding travel no matter how reasonable your letter is. But they may not.

One manager I talked to said this: “I’d find budget for some of my senior folks if they wanted to go [to a conference], just to keep them happy and engaged. [But] my teams haven’t asked.” (emphasis mine)

You’re not going to get fired for suggesting a professional development opportunity. The worst your boss can say is no. And you already have a no. So there is only upside.

I would love to see you in Las Vegas for ETLS next month. Again, you can use the code REVOLUTION to shave several hundred dollars off the ticket cost. Can you do two things for me? 1) if you follow the three questions above and feel ETLS or another event would be useful for you, can you generate the pitch letter and send it to your boss? and 2) can you let me know how it goes?

I’m very interested to hear your results.

Good sponsored thing

Reminder that Pluralsight is still running their 30-day trial of their learning platform for anybody working on the Cloud Resume Challenge. Use this link to claim your free month. Go get a cert or something. You’ll be glad to have invested the time in yourself.

Just for fun

If we all have to return to the office anyway, we might as well demand some training budget.

Jared’s words follow:

A comical application of AI and really fast search

I spent a couple days thinking about what to build, and after revisiting the Typesense feature list a few times, one idea stood out: semantic search using embeddings.

Quick primer on embeddings (skip this if you already know)

If you're not familiar with embeddings, they are a technique to represent data, especially complex or high-dimensional data, in a simpler, lower-dimensional form.

In the context of text, this means words that have similar meanings are mapped closer together in the vector space. For example, words like "vehicle,", "airplane," and "helicopter" would be placed nearer each other in the vector space, while they would be further from less related words like "pencil" or "telephone." This makes it easier for a machine to understand and process the relationships between different words. Let’s assume a 2-dimensional vector space (X, Y). Words like “vehicle” may be assigned a value like (2, 3) and “airplane” (2.2, 3.2) while a much less related “pencil” would be (5, 1).

Represented as a graph, vector search with those embeddings on the 2d space would look something like this.

You can see then how if you took similar words and a more dimensional space, most embedding models are hundreds or low single digit thousands right now. Searching for things like helicopters, we can use the dimensions to know that semantically, results like “transportation” are more relevant than “office”.

The project: “Relevant Brazeal”

So I knew I wanted to use embeddings to search for something, but I still wasn’t sure what. While I was procrastinating, it just so happened that Forrest had recently posted one of his classic comics…

I also had been looking for one of his comics that I knew had a Titanic or something on it describing the hidden complexities of serverless computing. There we go. I wanted a semantic / ludicrously fast search over Forrest’s comics.

I also gave myself a side quest… since we are leaning into AI, just lean all the way in. I wanted to hand write as little code as possible to see for myself how far we really are these days with the whole “AI coder” thing.

I got permission from Forrest, he dumped me a JSON blob of his comics, and away we go.

Problem 1: Images might be worth 1000 words, but they don’t actually have words

The giant JSON blob mostly amounted to a title and link to an image. This turns out to be approximately zero use when it comes to any kind of search, let alone semantic search.

Well, I said we were going all in on AI. I popped open zed, my editor of choice because it is real fast and has pretty solid AI capabilities, and then prompted as follows:

> write a script that uses OpenAI gpt-4o to extract the captions and descriptions of a comic. 

It spit out some Python. I haven’t used Python for a couple years at this point. So naturally this was followed by:

> how do I setup a python project in 2024

 > what is poetry

> no, i mean python poetry

Eventually we got there. A few more prompts of “make this script take an argument of a file with a JSON blob with these properties”, and some fine tuning of the prompt of what parameters I wanted in the response, and we had some surprisingly solid results from nothing more than an image and fairly simple prompt.

Code / Input: GitHub Link

Output for this image:

{
	"description_environment": "The image is a parody of a Jeopardy game show with a game board and contestants.",
	"description_happening": "A person stands at a podium, asleep or tired, faced with a board of 'Engineering Jeopardy' questions. The board's categories are 'Design Decisions', 'Time Estimates', 'Tool Choices', 'Best Practices', and 'Performance', with corresponding monetary values. Many of the answers on the board are 'It Depends'. Three contestants are raising their hands frantically.",
	"description_humor": "The humor arises from the absurdity of engineering questions often having the vague answer 'It Depends', poking fun at the complexity and uncertainty in the field of engineering.",
	"caption": "ENGINEERING JEOPARDY; COSTS; DESIGN DECISIONS; TIME ESTIMATES; TOOL CHOICES; BEST PRACTICES; PERFORMANCE; IT DEPENDS; $200; $200; IT DEPENDS; $200; $200; $400; IT DEPENDS; $400; $400; IT DEPENDS; IT DEPENDS; $600; $600; $600; IT DEPENDS; $600; IT DEPENDS; $800; $800; IT DEPENDS; $800; $800; IT DEPENDS; IT DEPENDS; $1000; $1000; IT DEPENDS; $1000; $1000; @forrestbrazeal",
	"tags": ["engineering", "jeopardy", "parody", "humor", "complexity"],
	"suggested_alt_text": "A parody of Jeopardy called 'Engineering Jeopardy'. The board features categories like 'Design Decisions' and 'Performance' with many answers being 'It Depends'. A tired contestant stands at a podium while three contestants raise their hands eagerly.",
	"title": "Engineering Jeopardy"
  }

I was impressed, to say the least.

• It had taken me maybe 15 minutes to get to this point.

• I wrote zero original code so far. I did edit the prompt a bit to add some additional properties to the response and give a little bit of extra detail in what I expected (captions to be separated by a semi-colon but still be a single string).

• Heck, I didn’t even pick the language the script was written in.

When I ran this, I realized it was going to take a little while to get a couple hundred of these images processed… so I simply prompted again:

> processing images is a bit slow, parallelize this to do it in batches of 10 and run all images in a batch in parallel. 

> add progress tracking and time estimation so I know how long this will take

On the first run, it imported some random lib and the script was crashing. Pretending I didn’t know why, I popped the error code in the chat assistant, it told me what to do and all was fixed.

Is it production-ready quality code and something I would be proud of? No. Does it work and do what I want? Seems like it. Plus it’s a script for something I wanted to tinker with. It’ll do.

Outside of the search usefulness, the AI also produced fairly good suggested_alt_text which could be useful for folks using screen readers to still get some of the description and possibly humor of the comic.

Problem 2: We have the data, now how do we search?

Popping open the Typesense documentation and searching (unsurprisingly powered by Typesense) for how to run it locally, there are a few options, and I opted for a quick docker run. A few moments later I had a local running Typesense instance. Refreshingly simple, and it also means I should be able run this on any container hosting platform and get an identical experience.

Now a story as a series of prompts:

Creating the script to index the collection

> Write a script that takes a JSON file path as an arg, gets the content, and writes it to a Typesense collection

> Make tags a faceted property

Now at this point, it worked. Simple curl requests to the local endpoint worked, but only for words really found in the text. Looking for something like “titanic” didn’t return anything. This makes sense; I hadn’t yet enabled the “Hybrid” search mode of Typesense, which takes both a keyword and semantic search together to produce the best results.

The weakest part of this process was that Typesense doesn’t really provide much guidance on the “best” embeddings models to use or much guidance on any of the differences in their docs. I assume this is largely because this is an “it depends” type situation on what you should pick, depending on your text domain.

I was at a total loss for which model to use. After some quick searching and finding the Massive Text Embedding Benchmark, I ended up going with ts/gte-small since it seemed like a good tradeoff of size vs quality. I have no idea if that was a good choice or not, but the results will speak for themselves.

Enabling the embeddings to be computed was incredibly simple, just a simple edit to the collection creation script (I did do this manually instead of prompting) and then recreating the index and we were ready for semantic searches.

  'fields': [

         // … all the other fields

         {

             "name": "embedding",

             "type": "float[]",

             "embed": {

                 "from": [

                 "description_humor",

                 "description_environment",

                 "title"

                 ],

                 "model_config": {

                 "model_name": "ts/gte-small"

                 }

             }

         }

    ]

  Creating the search “app”

This is what web app development looks like now:

> Write a single index.html file that uses instantsearch.js and typesense to provide a search over a collection, provide some basic clean styling and present results as a grid of images using the `url` property

> Add the title property to the top of each result item

> Add an instantsearch faceting component for the tags property, place it to the left of the image grid

> Provide a dark mode that respects the user browser / os settings

> Use the Cadillac of frontend, jQuery, to make each image grow to a reasonable readable size on hover

> Make each image click go to the reference url in a new tab

> Add a title and sub-header tags to the top of the page

I did make a few edits in the fairly readable produced code, like adding the title and links / info to the headers.

The ridiculous results

It turns out this all works better than I ever expected.

Not only was I able to fulfill my mission of finding the “comic with a Titanic in it”, but the semantic search is extraordinarily flexible. “Bird” picks out comics with chickens and penguins in it. “People waiting in line” picks out exactly what you’d expect. More exacting caption search is perfect. All of this searching works as quickly as I can type into the search bar, with results returning in milliseconds thanks to Typesense.

I threw everything up on GitHub Pages and you can play with it to your heart’s content. Once again, Typesense was kind enough to sponsor hosting the collection in their Typesense Cloud offering so I didn’t need to worry about managing anything to keep the demo alive long term.

You can also view all the source code here. Again, this isn’t production-quality code, it’s probably barely even re-usable code. But that wasn’t the point of this exercise!

I ended up walking away fairly impressed with the current state of things in AI, as well as the speed, ease of use, and developer-friendliness of Typesense. Starting from a JSON blob with a list of URLs to comic images, I was able to get incredibly fast, semantic search over that set of images, with minimal coding required from my end. All with a couple hours of prompting and reading some docs here and there.

So what’s the takeaway? Largely, I believe there is value for a developer to spend their time to push the bounds of what they think Gen AI can add to their workflow. Pick a toy project like this one, and challenge yourself to get as far as you can with prompting and iterating, without hand-jamming much code. Find the limits of what Gen AI can and can’t do (yet), and maybe discover some responsible ways to augment your day-to-day workflows.

Forrest here again. Thanks, Jared!

If you would enjoy getting sponsored to hack on interesting side projects like this, my cofounder Emily and I would love to hear from you.

Links and events

• Speaking of generative AI, I recently appeared on Rob Collie’s Raw Data podcast, trying to work out my complicated feelings about AI.

• Still speaking of generative AI, because aren’t we all, I will be premiering an absolutely ridiculous mini-musical about the types of ways AI makes us feel at Gene Kim’s ETLS event in August. I’ll be bringing a grand piano and all sorts of other nonsense to the Fontainebleau in Las Vegas. It should be … an experience. You should convince your boss to buy you a ticket. Say it’s for training.

• On Monday I closed out Day 1 of the marvelous fwd:cloudsec conference with some brand-new songs about RTO, AI, and other things that make us sad.

• Pluralsight, the longsuffering sponsor of the Cloud Resume Challenge, asked me to tell you about a new offer for challengers: if you post about the Cloud Resume Challenge on LinkedIn using hashtag #CloudResumeChallenge, and you tag Pluralsight as your learning platform, they will reach out to give you $50 in free AWS credits. I’m also told that if they really like what you said, they might even give you $1000 in credits. This seems like a pretty good offer to me. You should take them up on it.

Cartoon of the day

All this talk about Typesense and cartoons got me thinking…it’s time to bring back the Probably Wrong Flowcharts.

First up: In last week’s issue, I came after Google Cloud pretty hard for not posting a root-cause analysis of the mysterious outage that took down a large customer named UniSuper.

Credit to them, just a few days later they released a statement containing everything I called for and more:

• They took full responsibility for the issue without making excuses (even though, given that VMware Engine was indeed involved, I guess they could have tried to blame Broadcom).

• They provided a clear technical explanation of why the outage happened. That tricky word “subscription” that confused so many of us in the initial reporting about the issue? It’s referring to their provisioned VMware tenant, which was accidentally set to expire after one year.

• They laid out in reasonable detail what steps they are taking to prevent this class of incident from occurring in future, and why it was a freak occurrence unlikely to happen again: this was a config mistake during a since-deprecated setup process involving a single resource, not reflective of an underlying issue with GCP’s technical stability. Customers and investors can breathe a huge sigh of relief.

We can nitpick about how it would have been nice to have this statement within days rather than weeks, but I was hard on GCP last week so I want to be equally generous to them here. However long it took to put together, this is an outstanding statement. It’s clearly worded, technically informative without overly exposing the customer, and appropriately reassuring without being glib. I’m impressed by the work of the comms team, the marketing team, and of course the engineering and support teams who put it together.

I hope this is a template for the kind of communication Google Cloud will have with its customers going forward.

Oops, all cartoons

I’ve gotten a little behind on sharing new cartoons with you - so here are several, as well as a small announcement you may find interesting.

I’m not sure we’ve ever talked in this newsletter about why I draw so many weird little cartoons and flowcharts. Originally I drew them with sharpie marker in my cubicle at Infor and scanned them using the office copier. Now I use an iPad, but they still help me to organize my own thoughts, like about the misaligned incentives of open source development:

Anyway, I have been drawing them for years and years, there are hundreds of them now covering all sorts of tech topics, and hardly a day goes by that I don’t get requests for permission to use them in slide decks and LinkedIn posts and conference talks and O’Reilly books and on and on and on.

I’ve always been happy to give permission for folks to use the cartoons on a one-off basis, but a couple of additional needs have come up lately:

• Increasingly, the marketing teams at tech companies want commercially-licensed, high-res versions of the cartoons that they can use in their own decks and posts &c

• The cartoons do sometimes get out of date, and so I need a canonical place to post and let people know about updated versions

I don’t think I’ve mentioned it yet in this newsletter, but to keep up with these needs I recently added a subscription option to the Good Tech Things site. For $5 per month, you get full access to download royalty-free, up-to-date versions of the full cartoon archive that you can use however and wherever you want. You’ll also be the first to know via email whenever a new one is published. Early responses from both individuals and marketing teams have been very positive, so I’m letting y’all know about it in case it’s helpful to you as well.

One other question I hear every so often is “Are you going to open up a paid subscription option for your newsletter? Because I’d be happy to kick you a few dollars.” I’ve resisted doing that because I really want to reach as many people as possible with what I write, and putting it behind a paywall just feels kind of anti-internet to me. BUT - if you are one of those people, and would like to support Good Tech Things, a GT2 Pro subscription is the best way to go. And I hope you’ll find it useful in growing your own audience as well.

On to today’s Good (and not-so-good) Tech Things…

Today we’re looking at two concurrent issues that affected customers on AWS and Google Cloud. The way the providers responded reflects deep differences in their DNA .. and in their trustworthiness.

A tale of two clouds

AWS isn’t what they used to be. “It’s feeling a lot like Day Two over here,” friends at AWS often tell me, shaking their heads sadly. A sweaty pursuit of generative AI has clouded their historic focus on IaaS building blocks. A myopic insistence on RTO continues to force out top talent. They’re even deprecating services now. (My 168 AWS Services song from 2020 references at least 4 services, not counting SimpleDB, that have since been or soon will be entirely scrubbed from the docs and SDKs: Workdocs, Honeycode, Snowmobile, and Alexa for Business.)

But you know what AWS still does better than anybody else? They actually listen to their customers and bend over backwards to help them out.

Case in point: on April 29th, a software engineer named Maciej Pocwierz discovered an unusual behavior in Amazon S3 that left him on the hook for a $1,300 bill.

Cliff’s Notes explanation of the behavior:

• S3 buckets all exist in a single global namespace.

• Anybody can try to access any S3 bucket, with or without an AWS account, if they happen to guess the bucket name.

• Assuming the bucket is not publicly accessible, the unauthorized user will get a generic HTTP 4xx error code.

In this case, the unauthorized requests (over 100 million of them!) were coming from the makers of a widely-used open-source tool, who had accidentally shipped a default configuration that collided with Maciej’s bucket name. They quickly rolled out a fix, and the problem stopped.

So far, so reasonable. What surprised Maciej was that each of those 100 million unauthorized requests gets billed to the bucket owner. In other words, it’s possible to perform a “denial-of-wallet” attack simply by spamming bad requests to your enemy’s bucket. And even though AWS Support was kind enough to refund this particular bill, there was nothing Maciej could do to protect against this attack in future. It was just a built-in risk of using S3.

Now, there were several ways AWS could have responded once Maciej’s blog about the incident gained wide attention.

1. They could have said “This is expected behavior; just use randomized bucket names that are harder for attackers to guess.”

2. They could have pointed out that it’s quite resource-intensive to spin up enough of these requests to make a meaningful dent in enterprise wallets, so this probably isn’t going to be a big risk for big customers.

3. They could have just stayed quiet about the whole thing and waited for everyone to forget about it.

But they didn’t do any of that. Instead, AWS’s official Voice of the Customer, Jeff Barr, acknowledged within 24 hours that people shouldn’t have to pay for unauthorized S3 requests they didn’t initiate. Within a week, the S3 team was working on a fix. And as of May 13th, customers will no longer incur request or bandwidth charges for HTTP 403 requests initiated from outside their own environment.

I just want to pause for a second on how remarkable this is. S3 is an enormous, mature product with all sorts of complex billing dimensions. Although they employ a lot of engineers, I’m quite sure everyone was busy with a full roadmap of other priorities for the quarter. And yet … they found time to roll out a nontrivial behavior change based on the complaint of one person at a small Polish consulting company who had incurred a total bill of $1300 US dollars. Elapsed turnaround time from complaint to rollout was 2 weeks.

That, my friends, is what customer obsession looks like. As long as AWS can still do that, they are the cloud to beat.

Meanwhile, in the other cloud…

At the exact same time all this was unfolding, Google Cloud was having troubles of its own.

On May 2nd, the CEO of a $135 billion Australian pension fund called Unisuper acknowledged a service outage affecting its 647,000 customers—many of them retirees whose life savings are tied up in the company. The whole thread of outage updates is worth a read. It starts out fairly generic and, as the days go by with no resolution in sight, becomes increasingly bewildering. Unisuper hastens to assure its members that ransomware is not involved, that nobody’s data is at risk … but that getting all this resolved is going to take quite a bit of time, and apparently Google Cloud is right in the middle of it all.

On the 8th of May, a full week into the outage, Unisuper issued a bizarre statement on its website purporting to be from both their CEO and from Google Cloud’s CEO Thomas Kurian. The statement placed full responsibility on Google Cloud for “deleting Unisuper’s private cloud subscription” and said that the only chance Unisuper had to get things back online was that they had squirreled away some backups on an entirely different cloud.

This statement was so weird that I, among others, immediately went on record with doubts. For one thing, Google themselves had been completely silent on this. For another, there’s no such logical boundary in Google Cloud as a “subscription”; that’s Azure terminology. And in the ~3 years I was at Google I never once heard Thomas Kurian make a personal public statement taking the blame for a customer outage. The whole thing sounded suspiciously like Unisuper was making things up to deflect blame for their $135 billion fumble.

But then—incredibly—Gergely Orosz was able to get confirmation from Google Cloud PR that the Unisuper statement was legit. Google Cloud really did delete one of the the biggest pension funds in Australia, and left their customer to handle all the fallout.

Questions arise.

In a statement presumably lawyered to death by both companies, why did Google use conspicuously inaccurate language to describe its own services? The only thing I can come up with is that Google Cloud preferred not to reveal exactly which of their services ate Unisuper. Given the oblique reference to a private cloud, we can speculate it might have been VMware Engine. But other customers of that service have no way to know for sure.

How, exactly, does a public cloud accidentally delete a major customer so irrevocably that it takes them eleven days and off-cloud backups to get back online? I do not know. I do not know how this is possible. It raises all sorts of unpleasant questions about what other safeguards Google Cloud’s ops team is missing behind the scenes. This is the sort of problem that cries out for a full, detailed, public root-cause analysis. Don’t delegate to your customer to post a vague assurance that “this was a one-off mistake and we really super promise it won’t happen again.” This is a bad, bad, bad problem. A confidence-shaking, existential problem. You have to make a direct statement demonstrating that you know what went wrong and why it won’t happen again.

But Google Cloud didn’t. They usually don’t. And customers yet again are left wondering, waiting, and considering if they should make off-cloud backups of their own.

That is the opposite of customer obsession. It is customer obfuscation. If Google Cloud wants to be taken seriously as a competitor to AWS, they should learn the lesson of the S3 denial-of-wallet attack. Customer trust is not earned through shiny AI demos and inexplicable musical performances. It is earned the hard way, one support case at a time.

Cartoon of the day

Today’s cartoon is brought to you by Tidelift. They’d love to see you at Upstream, their conference digging into the health and security of open source on June 5.

I do remember that I felt nervous, overwhelmed, out of my depth in my new job. I was too embarrassed to ask a coworker for help. Posting on Stack Overflow felt like a last-ditch effort, a Hail Mary pass. The next day I logged in and checked my post, hoping against hope that somebody had taken pity on my plight.

The post had been closed as off-topic.

I didn’t post again for years.

Even a decade ago, Stack Overflow had already developed a reputation as kind of a prickly, intimidating place. Users weren’t shy to let you know if they thought you had posted something dumb. The top contributors (like the legendary Jon Skeet) were so fast at gobbling up answers to the good questions that it was hard for new contributors to amass reputation points. The system of relying on user upvotes to rank multiple answers, rather than crowdsourcing a single canonical response to each question à la Wikipedia, meant that out-of-date or just plain wrong answers often appeared at the top of the page.

None of it mattered because Stack Overflow was a quantum leap in surfacing help for programmers. The upvoting system was janky, but it worked better than any previous tech forum. (Remember “Tom’s Hardware”? The 27th page of Google search results sure does.) The userbase was polyglot and omniplatform: you could find answers on just about any programming topic. And it was certainly faster than reading man pages. Or, crucially, it felt faster.

Stack Overflow gave stumped programmers the dopamine hit of looking up the solution to their problems, rather than learning how to solve them.

In this way, too, Stack Overflow was ahead of its time.

You know something weird? When Legendary Jon Skeet passed one million reputation points in 2018, Stack Overflow celebrated by creating a Markov chain bot that spit out Skeet-shaped answers to arbitrary questions. At the time everyone seemed to think this was harmless fun.

In retrospect, it looks like foreshadowing.

LLMs have not, so far, proved to be as generally useful as a lot of true believers hoped. They’re bad at surfacing insights not found in their training data, they’re asymptotically mediocre at “creative” tasks, and they’re untrustworthy in any field requiring accurate information.

But they have turned out to be reasonably accurate at producing answers to common programming questions. And GitHub Copilot and ChatGPT are much faster and easier to use than navigating the labyrinth of a question-and-answer site.

Most importantly, the thing Stack Overflow was worst at — providing a welcoming place to learn — is the thing LLMs are the best in the world at. Whereas posting on Stack Overflow can be like releasing a fragile bird into a cage full of furious snakes, ChatGPT does not care if your question is a duplicate. It does not care if you did not provide enough context. It is not hung up on parliamentary procedure. ChatGPT is infinitely patient and (more or less) constantly available.

My younger self, two days into a job I barely understood, would have greeted ChatGPT like the messiah.

The people who work at Stack Overflow are not stupid. They understood immediately that LLMs were an existential threat to their business. The problem is what to do about it. They’ve tried launching their own AI tools, but it’s not clear that they’re better than any of the other 500,000 coding-help models out there.

Instead, they’ve just announced a wide-ranging partnership with Open AI. These partnerships are sometimes public relations niceties more than they are substantive collaborations, but it does sound as though Stack Overflow is going to license its data for Open AI to train future models on.

It’s hard not to see this as anything other than an attempt by Stack Overflow’s newish corporate overlords to cash in on their intellectual property before it becomes entirely worthless.

What remains of the Stack Overflow community is disgusted by the idea that their contributions will be absorbed into anonymous AI models, perhaps devaluing their hard-won reputation points and community credibility. Some people are trying to delete their own questions out of spite, only to discover that they don’t actually own their own content. And I don’t know why anyone would willingly submit more unpaid answers (or questions!) to Stack Exchange forums under these conditions. After all, huge pools of people in Africa and South America are being paid to write code examples for model-training companies right this minute. Why should I do it for free?

When LLMs first became available, a whole bunch of people tried to use them to generate Stack Overflow answers and rack up cheap karma; the overall quality of answers, though, was low enough that Stack Overflow has banned AI-generated content entirely from the platform.

This raises the most troubling problem going forward, not just for SO but for all of us. The relationship between user-generated training data and AI-generated results so far appears to be one-way. LLMs do not add to humanity’s body of knowledge; they only synthesize and regurgitate.

Particularly in the programming world, where many questions are niche, context-specific edge cases, the only way to get high-quality answers is through a lot of net-new human experimentation. Stack Overflow, to its credit, built a community that incentivized this type of knowledge sharing.

It is not obvious that LLMs will do the same.

A coda: I discovered today a whole trove of questions on Stack Overflow related to the Cloud Resume Challenge. Beginning programmers, just like I was all those years ago, are crying out for help—stumped by DNS SERVFAILs, by Python path woes, by responsive CSS. And in almost every case, they are getting help. Real advice from real people, who are not just showing them how to solve their problems, but leaving a public record for the next new programmer to find. I saw little if any judgmental behavior—the community has matured over the years. I saw encouragement, respect, and growth.

Despite all its warts, perhaps even because of them, Stack Overflow has carried the torch for an extraordinarily human way of exploring technology. Open AI would do well to consider how they could put more of that into the world, rather than less.

Good Sponsored Thing

I’m super excited to announce that Pluralsight, a leading provider of tech courses, labs, and cert prep, has signed on as the first official sponsor of the Cloud Resume Challenge.

What does this mean for the CRC? I've always said that if the Cloud Resume Challenge community were to work with sponsors, it would have to be in exchange for something that would clearly benefit the community.

I believe Pluralsight has met that bar. They are offering 30 days free on their Premium plan for all Cloud Resume Challengers. This is 3x longer than their standard personal free trial, and should give you enough time to study for an intro-level cert without having to worry about finances.

The link to the free 30-day offer can be found on the CRC challenge pages, or you can redeem it directly here.

I hope this is helpful to many of you - feel free to share the redemption link with friends, and if you want to thank Pluralsight, I'm sure they'd be thrilled if you tagged them in a social post.

Cartoon of the day

The US Department of Homeland Security’s new AI Safety and Security Board is stocked with a remarkable assortment of the clueless and the self-interested.

IBM has become such a shadowy afterthought1 that I don’t think too many people have strong feelings about them anymore. The big question is whether they will ditch HashiCorp’s new BUSL licensing scheme and go back to a true open-source model for products like Terraform and Vault, and I think it speaks to people’s lack of a handle on what IBM is up to these days that a plausible sentiment is “hey, they might do it!”

The main reaction from the community about HashiCorp spinning an increasingly narrow set of options into a $6.4 billion exit is “Welp … good for them.” And when a community says good for them, it’s quite conspicuous what they’re not saying: good for us.

But I want to raise a different question today: how the heck did we get to this point? Why was HashiCorp not snapped up long ago by a more viable cloud provider? Specifically: why did they not get bought by Google Cloud?

On the surface it seems like a match made in heaven, doesn’t it?

• Google Cloud is the number three cloud provider; a huge slice of their competitors’ customers use and love HashiCorp products every day. Seems like a chance to steal some new fans.

• Google has historically cultivated a generous attitude toward open source. Hear those drums in the distance? That’s the cursed board game of Kubernetes floating down the river towards you right now. I bet they wouldn’t think twice about donating Terraform to the Linux Foundation, bringing the renegades at OpenTofu back into the fold.

• Google Cloud has a long history of making large strategic ops acquisitions (Stackdriver, Mandiant, Chronicle). Adding Vault and friends to their product portfolio would not be out of character for them.

• Google Cloud already treats Terraform pretty much as their default deployment option! (Technically they have a bespoke service called “Google Cloud Deployment Manager”, but it is deprecated in all but name and even GCP’s own evangelists will tell you to just use Terraform instead.)

• I bet, if Google looked real hard under all the couch cushions, they could come up with 6.4 billion dollars.

These are all superficially compelling reasons to go out and buy HashiCorp.

But the reasons not to buy it are far more compelling—and, I think, underscore what a tough spot HashiCorp was in, and why I’m sure their reaction to signing the LOI with IBM was a huge sigh of relief.

Why buy the cow when you can get the milk for free?

This is the cynical reason. Google Cloud is grabbing close to 100% of the value of Terraform today from the OSS version. If they want a hosted Terraform service, they can just build one themselves.

And, in fact, that’s exactly what they did. Last fall Google Cloud announced Infrastructure Manager, a service that slips managed Terraform deployments right into your existing Google Cloud CI/CD pipelines.

I was at Google when this rolled out, and a product manager on the service told me at the time that Infrastructure Manager really wasn’t intended to compete with HashiCorp Cloud Platform—after all, it was a very Google Cloud-specific solution designed to make Terraform feel like a seamless part of the GCP deployment experience.

If you’re HashiCorp, of course, that’s the last thing you want: for Terraform to become an invisible substrate of the cloud providers. I happen to know that HashiCorp was NOT happy about the release of Infrastructure Manager. But there wasn’t much they could do other than change their license model.

$6.4 billion is just the beginning

But let’s say Google buys HashiCorp anyway, just for the PR bump and the sales talking points. That’s when you realize that $6.4 billion is just the cover charge, not the full tab.

HashiCorp Cloud Platform (the hosted Terraform, Vault, etc that HC manages themselves, their nominal secret sauce) runs on AWS today. So now you’ve got to spend time and money on a huge migration, probably multiple years of planning and work.

Also, to sell HashiCorp products into large enterprises you probably need to fund lots and lots and lots more enterprise support and solutions architects, the competencies Google Cloud is persistently the worst at.

Oh yeah, plus—you’re already expending resources on the Google Cloud Terraform provider. But now you’re stuck having to support and co-maintain providers for a pile of other partners, including your biggest competitors. You are effectively paying for AWS and Azure to use your $6.4 billion dollar toy for free.

IBM might be willing to take that deal if it means a snowball’s chance of being taken seriously as a for-real cloud provider, but I don’t think it really makes sense for Google Cloud.

Which brings us to the existential question:

Do you really, really want to be in the “hybrid cloud deployment” business?

Like, what’s the strategic value for Google Cloud in being the Terraform fairy? People with AWS footprints aren’t going to use Google Cloud to manage state for their deployments, that’s just weird. Enormous companies have always preferred to roll their own infra platforms, meaning your real customers for managed Terraform, Consul, etc, are mostly mid-sized companies who have obnoxious price sensitivity. You’re taking on all the responsibility of keeping up with a notoriously fast-changing set of infrastructure tools while getting very unclear revenue potential in return.

Heck, I don’t even think HashiCorp wanted to be in the hybrid cloud deployment business. Their recent public filings make it clear that they couldn’t figure out a SaaS business model and weren’t confident in their ability to become a services business.

I think the only real argument for Google buying HashiCorp would come down to brand value and OSS goodwill. And as they’ve recently indicated by firing all their Python maintainers, hard-to-quantify efforts to improve the OSS commons don’t carry the weight at Google that they once did.

We’ll never know for sure, but I think those are the big reasons HashiCorp was never acquired by a major cloud provider.

Anyway, HashiCorp eventually found a buyer that made sense, all these problems are IBM’s problems now, and hopefully the people who built one of the most truly remarkable and useful open-source software companies of all time can get a well-deserved night’s rest or two.

In all sincerity: good for them.

Cartoon of the day

I, too, have been through an acquisition.