The Sleep-Promoting Resilience of Boring Infrastructure

In last month’s post on incremental improvements, I highlighted some recent user-facing updates we’ve made here at Rogue Amoeba. Today, I’d like to share some details on the backend systems that have kept the company running strong for nearly a quarter of a century.

Measured against current development trends, Rogue Amoeba takes a very conservative approach to infrastructure. You might guess this is because we’re not a web app company, so we don’t have to think about infrastructure at scale. While there is a measure of truth to that, our infrastructure codebase is actually nearly double the size of the codebase for our commercial products. That infrastructure powers a wealth of systems, including our online store, Customer Dashboard, software updates, and Support Center.

Because of the importance of those systems, we’ve long avoided adopting untested technologies and complex architectures. A few weeks ago, while discussing the fact that one of our most recent rollouts found us “updating” to a 30-year-old technology, I jokingly asked our CTO Quentin, “What’s the opposite of being on the bleeding edge?” His response echoes one of the core tenets of our infrastructure philosophy:

“Being resilient.”

In our infrastructure planning, we regularly ask how we can make things less complicated, easier to maintain, and more resilient. When we make a change, ensuring our customers can access our site, download and purchase our software, and get the support they need is top of mind.

At a past job working in real-time transaction processing for a major fintech, I received the advice to deploy systems that won’t keep you up at night. That’s what we do at Rogue Amoeba. Most of our infrastructure decisions focus on decades-old, battle-tested technologies. Further, whenever we do update a dependency or migrate tech, those changes are rolled out slowly and deliberately.

And so, here are two recent examples of sleep-promoting systems we’ve deployed atop “boring” tech stacks:

FastCGI Migration

After we enhanced our software update system to show richer release notes, we noticed the changes were putting a relatively high strain on our server. That led us to take a hard look at some of the most legacy of our legacy systems, our CGI scripts. Over the years, we built these to handle things like update notifications, in-app support requests, and order fulfillment. With the Python foundation recently having removed the cgi library1, we decided it was time to start looking at other options.

There are many modern tech stacks we could have chosen to migrate these systems to, including AWS Lambda, Google Cloud Functions, and Cloudflare Workers. But instead, we asked, “What’s the simplest thing we can do to make these scripts significantly more performant?” That question was answered about 30 years ago, with the development of FastCGI.

After some initial testing and experimentation, we migrated all of our CGI scripts to FastCGI relatively easily. The performance improvement was notable, making it clear that “FastCGI” is named well:

This graph compares four performance metrics for one of our most heavily utilized APIs, across both the old CGI stack and the new FastCGI stack. “Mean” represents the average response time of all requests, and while that improved significantly, the worst-case scenarios are especially striking. To measure those, we used the common P90/P95/P99 SLA metrics.2, which say “90%/95%/99% of requests are handled faster than this.”

As you can see above, all three of those worst-case scenarios improved significantly under FastCGI. Could AWS Lambda have gotten us down to single-digit response times? Maybe. Would it have exponentially increased our infrastructure complexity and cost? Definitely. For us, that trade-off wasn’t worth it.

SQLite Migration

Another recent migration involved transitioning our primary order database from MySQL to SQLite. Folks who know about databases might think I wrote that backwards, but no.

Industry thinking around databases has shifted quite a bit over the years. By the early 2000s, MySQL and PostgreSQL had become the standard databases used in web development. That was followed by a brief surge of “NoSQL” advocacy that warned “you’d better be ready for big data.” As a result, many people migrated to systems like Mongo and Cassandra.

I have a good amount of experience with big data, having worked for both a major bioinformatics research institute that dealt with petabytes of genetic data3 and a fraud-analytics company that had to process billions of rows of transactional data. While database engines like Mongo and Cassandra have their place, the majority of people who adopted those systems are wielding way more database power than they need. A data engineer friend of mine once remarked that powering most websites with Cassandra or Mongo is like buying a fully loaded SUV to drive to the corner store.

While MySQL and PostgreSQL were dominating the web application world, around the same time period a smaller, lesser-known database system called SQLite had begun taking over the embedded systems space. You’d probably be surprised by the number of things in your life that are powered by SQLite databases. It is now the most widely used database system in the world4, powering cell phones, automotive systems, web browsers, modern planes, and even satellites.5

Unlike larger database engines that require dedicated server setups and separate permission sets, SQLite databases consist of a single file that can sit alongside the rest of your web application code. Despite their simplicity, on modern hardware using WAL6, SQLite can easily handle tens of thousands of reads and writes per second. On more sophisticated hardware with software-level optimizations, it can go orders of magnitude beyond that.7 Since SQLite is a file-system-based DB, permissions are a breeze to manage, and developments like LiteFS make replication just as straightforward.

The best part is you can back up and restore your entire infrastructure and data stack with a simple file copy, or use tools like Litestream to stream changes into object storage.

SQLite databases already powered most of our internal systems at Rogue Amoeba, but the order database was a holdout, still relying on MySQL. We knew migrating it would be a long-term win, but the migration involved extensive code changes. Further, since these systems handle order fulfillment, the method and timing of the migration were critical as we didn’t want to deploy multiple sweeping changes simultaneously. We moved slowly, rolling out the necessary changes in heavily tested phases. Eventually, we ran both database systems in parallel for a time, until we were satisfied that the new system performed well.

The final result has exceeded our expectations. Performance is spectacular and DB operations like schema changes now complete in milliseconds, compared to the sometimes hours it took with MySQL. Best of all, we have one fewer set of servers, permissions, and network configs to manage.

Conclusion

The term “legacy code” is often used in a negative light, as something that needs to be replaced simply because of its age. But with so many recent stories about AI-deleted codebases, poisoned NPM dependency graphs, the left-pad fiasco, and cloud outages that take down half the internet, I prefer the other definition of “legacy”: a long-lasting impact of events and actions.

Our infrastructure systems are not exciting, and we’re just fine with that. Keeping our infrastructure “boring” and occasionally embracing battle-tested, “legacy” technologies allows us to keep our focus on our true legacy: building award-winning software that improves the lives of our customers.

Footnotes:

See PEP 594 for a discussion on the rationale for removing cgi support from the Python standard library. ↩︎

SLAs, or “Service-level agreements”, are the set of performance and uptime promises service providers guarantee to customers. P90, P95, and P99 thresholds are often part of those agreements. ↩︎

If you need to handle really big data, take a look at that research facility’s ITS infrastructure blog for some tips. ↩︎

See SQLite’s deployment stats page for details. ↩︎

For more fun examples, click here and here. ↩︎

WAL stands for “Write-Ahead Logging”, which allows SQLite to append transactions to a separate log file, then commit them in batches. It’s not only faster, it also prevents readers and writers from blocking one another. For more details, see https://sqlite.org/wal.html. I suppose I should note that WAL mode adds two more files, alongside the main database file. ↩︎

Here’s an impressive example of someone reaching 4 Million queries per second with SQLite. ↩︎