Under The Microscope

The Sleep-Promoting Resilience of Boring Infrastructure

“Legacy” doesn’t have to be a bad word.

In last month’s post on incremental improvements, I highlighted some recent user-facing updates we’ve made here at Rogue Amoeba. Today, I’d like to share some details on the backend systems that have kept the company running strong for nearly a quarter of a century.

Measured against current development trends, Rogue Amoeba takes a very conservative approach to infrastructure. You might guess this is because we’re not a web app company, so we don’t have to think about infrastructure at scale. While there is a measure of truth to that, our infrastructure codebase is actually nearly double the size of the codebase for our commercial products. That infrastructure powers a wealth of systems, including our online store, Customer Dashboard, software updates, and Support Center.

Because of the importance of those systems, we’ve long avoided adopting untested technologies and complex architectures. A few weeks ago, while discussing the fact that one of our most recent rollouts found us “updating” to a 30-year-old technology, I jokingly asked our CTO Quentin, “What’s the opposite of being on the bleeding edge?” His response echoes one of the core tenets of our infrastructure philosophy:

“Being resilient.”

In our infrastructure planning, we regularly ask how we can make things less complicated, easier to maintain, and more resilient. When we make a change, ensuring our customers can access our site, download and purchase our software, and get the support they need is top of mind.

At a past job working in real-time transaction processing for a major fintech, I received the advice to deploy systems that won’t keep you up at night. That’s what we do at Rogue Amoeba. Most of our infrastructure decisions focus on decades-old, battle-tested technologies. Further, whenever we do update a dependency or migrate tech, those changes are rolled out slowly and deliberately.

And so, here are two recent examples of sleep-promoting systems we’ve deployed atop “boring” tech stacks:

FastCGI Migration

After we enhanced our software update system to show richer release notes, we noticed the changes were putting a relatively high strain on our server. That led us to take a hard look at some of the most legacy of our legacy systems, our CGI scripts. Over the years, we built these to handle things like update notifications, in-app support requests, and order fulfillment. With the Python foundation recently having removed the cgi library1, we decided it was time to start looking at other options.

There are many modern tech stacks we could have chosen to migrate these systems to, including AWS Lambda, Google Cloud Functions, and Cloudflare Workers. But instead, we asked, “What’s the simplest thing we can do to make these scripts significantly more performant?” That question was answered about 30 years ago, with the development of FastCGI.

After some initial testing and experimentation, we migrated all of our CGI scripts to FastCGI relatively easily. The performance improvement was notable, making it clear that “FastCGI” is named well:

This graph compares four performance metrics for one of our most heavily utilized APIs, across both the old CGI stack and the new FastCGI stack. “Mean” represents the average response time of all requests, and while that improved significantly, the worst-case scenarios are especially striking. To measure those, we used the common P90/P95/P99 SLA metrics.2, which say “90%/95%/99% of requests are handled faster than this.”

As you can see above, all three of those worst-case scenarios improved significantly under FastCGI. Could AWS Lambda have gotten us down to single-digit response times? Maybe. Would it have exponentially increased our infrastructure complexity and cost? Definitely. For us, that trade-off wasn’t worth it.

SQLite Migration

Another recent migration involved transitioning our primary order database from MySQL to SQLite. Folks who know about databases might think I wrote that backwards, but no.

Industry thinking around databases has shifted quite a bit over the years. By the early 2000s, MySQL and PostgreSQL had become the standard databases used in web development. That was followed by a brief surge of “NoSQL” advocacy that warned “you’d better be ready for big data.” As a result, many people migrated to systems like Mongo and Cassandra.

I have a good amount of experience with big data, having worked for both a major bioinformatics research institute that dealt with petabytes of genetic data3 and a fraud-analytics company that had to process billions of rows of transactional data. While database engines like Mongo and Cassandra have their place, the majority of people who adopted those systems are wielding way more database power than they need. A data engineer friend of mine once remarked that powering most websites with Cassandra or Mongo is like buying a fully loaded SUV to drive to the corner store.

While MySQL and PostgreSQL were dominating the web application world, around the same time period a smaller, lesser-known database system called SQLite had begun taking over the embedded systems space. You’d probably be surprised by the number of things in your life that are powered by SQLite databases. It is now the most widely used database system in the world4, powering cell phones, automotive systems, web browsers, modern planes, and even satellites.5

Unlike larger database engines that require dedicated server setups and separate permission sets, SQLite databases consist of a single file that can sit alongside the rest of your web application code. Despite their simplicity, on modern hardware using WAL6, SQLite can easily handle tens of thousands of reads and writes per second. On more sophisticated hardware with software-level optimizations, it can go orders of magnitude beyond that.7 Since SQLite is a file-system-based DB, permissions are a breeze to manage, and developments like LiteFS make replication just as straightforward.

The best part is you can back up and restore your entire infrastructure and data stack with a simple file copy, or use tools like Litestream to stream changes into object storage.

SQLite databases already powered most of our internal systems at Rogue Amoeba, but the order database was a holdout, still relying on MySQL. We knew migrating it would be a long-term win, but the migration involved extensive code changes. Further, since these systems handle order fulfillment, the method and timing of the migration were critical as we didn’t want to deploy multiple sweeping changes simultaneously. We moved slowly, rolling out the necessary changes in heavily tested phases. Eventually, we ran both database systems in parallel for a time, until we were satisfied that the new system performed well.

The final result has exceeded our expectations. Performance is spectacular and DB operations like schema changes now complete in milliseconds, compared to the sometimes hours it took with MySQL. Best of all, we have one fewer set of servers, permissions, and network configs to manage.

Conclusion

The term “legacy code” is often used in a negative light, as something that needs to be replaced simply because of its age. But with so many recent stories about AI-deleted codebases, poisoned NPM dependency graphs, the left-pad fiasco, and cloud outages that take down half the internet, I prefer the other definition of “legacy”: a long-lasting impact of events and actions.

Our infrastructure systems are not exciting, and we’re just fine with that. Keeping our infrastructure “boring” and occasionally embracing battle-tested, “legacy” technologies allows us to keep our focus on our true legacy: building award-winning software that improves the lives of our customers.


Footnotes:

  1. See PEP 594 for a discussion on the rationale for removing cgi support from the Python standard library. ↩︎

  2. SLAs, or “Service-level agreements”, are the set of performance and uptime promises service providers guarantee to customers. P90, P95, and P99 thresholds are often part of those agreements. ↩︎

  3. If you need to handle really big data, take a look at that research facility’s ITS infrastructure blog for some tips. ↩︎

  4. See SQLite’s deployment stats page for details. ↩︎

  5. For more fun examples, click here and here. ↩︎

  6. WAL stands for “Write-Ahead Logging”, which allows SQLite to append transactions to a separate log file, then commit them in batches. It’s not only faster, it also prevents readers and writers from blocking one another. For more details, see https://sqlite.org/wal.html. I suppose I should note that WAL mode adds two more files, alongside the main database file. ↩︎

  7. Here’s an impressive example of someone reaching 4 Million queries per second with SQLite. ↩︎

Fifteen Fantastic Years

Our 15 year club has its second member.

It’s once again time to celebrate an employee milestone here at Rogue Amoeba. Membership in our 15 year club recently doubled, with Grant Farr joining that exclusive group. It is an honor and a privilege to celebrate him today.

Helming Audio Hijack

When we marked Grant’s five and ten year milestones, the primary focus was on Audio Hijack, and with good reason. From 2011 through today, his steady hand has guided our flagship to continued excellence. In 2015, we shipped Audio Hijack 3, the app’s revolutionary leap forward after over a decade of version 2. In 2022, Audio Hijack 4 shipped, an evolutionary update. In between, and since, we’ve had point updates that added new features and functionality.

Grant led the app through all of that, and he continues to lead it today. His work has made Audio Hijack remarkably stable, even as the app has evolved and changed, with year after year of steady improvements really adding up.

Audio Hijack is now up to version 4.5, and it remains one of the Mac’s most powerful and popular tools. Of course, there are more updates and improvements in the works too, and we know you’ll love what Grant’s been cooking up.

And Piezo

A decade and a half of leading Audio Hijack would be plenty, but that’s not the only audio recording tool Grant has worked on. Since 2011, he’s also served as the lead developer on Piezo, our charmingly simple recording app. Piezo was originally created to make recording as easy as possible, and nearly 15 years later, it’s still doing that. That’s possible in no small part thanks to Grant’s work, which created a dependable app that doesn’t require a lot of upkeep.

And Farrago 1

We can’t forget the third app Grant’s led here at Rogue Amoeba: Farrago. What began as an idea to include inside of Audio Hijack grew into the Mac’s best soundboard app. Development of Farrago was eventually handed off for version 2, but the app is still built on top of the wonderful work Grant did to create Farrago 1.

Even More

Grant’s contributions have been crucial to even more of our product lineup. Much of Grant’s user interface work is actually shared across multiple applications. Big parts of work he’s done have then been re-used in other apps, including Loopback and SoundSource. In ways both big and small, Grant’s work helps countless users every single day.

Take Your Pick

With Chris Barajas’s anniversary last year, we decided new members of Rogue Amoeba’s 15 year club would receive an original Apple watch:

The original Apple watch, close up

At the time, I noted that the timepiece in question dated to 1995 and Mac System 7.5. Long-time Apple fans might recall that this watch was specifically part of a promotion to encourage users to purchase an OS upgrade for their Macs:

An ad from 1995 offering the choice between two free gifts, a watch or Conflict Catcher 3, when purchasing System 7.5 for their Mac
[Image credit: Hodinkee]

We decided we’d one-up Apple’s promotional offer by providing both promo options. Yes, in addition to a beautiful and bizarre original Apple watch, folks who work at Rogue Amoeba for 15 years also earn a copy of Conflict Catcher 3:

Grant will need to locate his own floppy drive, however.

Cheers to Grant

For over 15 years, Grant has been writing rock-solid code for Rogue Amoeba. Thanks to him, Audio Hijack has been incredibly stable, and that’s crucial for a production recording tool. His work also led to the low-maintenance delight that is Piezo, gave a strong foundation to Farrago, and provided crucial parts of Loopback and SoundSource. Not bad!

The positive impact Grant has had on Rogue Amoeba can’t be overstated, and we’re so glad to have him on our team. Thank you for all you’ve done, Grant, and here’s to many more years working together!

SoundSource 6.1 Is Here

You’ll find dozens of improvements in this free update.

We’ve just released SoundSource 6.1, a free update that brings dozens of refinements and improvements to all SoundSource 6 users. Read on for more details, or just click to download it immediately.

Continued Progress

Last winter, we shipped SoundSource 6. It was a major upgrade to our popular audio control utility that brought a bevy of new features. Today’s update builds on that progress, with a few small new features, some big refinements and optimizations, and more than two dozen bug fixes.

More Ways to Access Quick Configs

SoundSource’s quick configs save complete audio configurations for your Mac, then allow you to switch between them instantly. We’ve just added two new ways to select your quick configs.

Most notably, you can now put your quick configs right in your menu bar, as seen to the right. Enable the optional menu bar icon with the switch found in the Manage Quick Configs sheet. As well, SoundSource’s alternate menu (accessed by Control-clicking the main menu bar icon) now has a Quick Configs sub-menu.

Access your quick configs, quicker than ever.

Clearer Volume Levels in the Menu Bar

This update brings another notable change to the menu bar, as we’ve improved the volume level indicators found there. The previous three dots in SoundSource’s menu bar icon have now been replaced with four wider levels.

This updated art is shown when SoundSource’s Menu Bar Icon (in the Appearance settings) is set to either the Speaker (SoundSource) or Match Output Device.

Refinements for Bluetooth, Audio Devices, and Audio Units

Bluetooth outputs can often cause headaches, but this update reduces the pain. We’ve eliminated several issues that could cause audio glitches and dropouts when playing to your Bluetooth devices.

Speaking of audio hardware, SoundSource’s Audio Devices window has also been updated. It now includes “Preferred Device Order” controls for MacOS’s “Sound Effects” output, in addition to ensuring that devices you mark as hidden correctly stay hidden everywhere.

If you use SoundSource to enhance audio on your Mac, we’ve got plenty for you as well. We made multiple improvements to Audio Units in version 6.1. Of particular note, newly installed effects are now instantly ready for use in SoundSource.

Substantial Optimizations

This update also fine-tunes the ARK plugin that powers audio handling, making it more efficient than ever. To minimize resource consumption, we worked around several issues in MacOS itself. Recent tests show CPU usage down 18-35%.

We‘ve also elevated the experience when using SoundSource to send to AirPlay devices. Optimizations in streaming code allow for improved synchronization between AirPlay outputs, along with better connection reliability.

Bugs, Squashed

Bugs be goneWe’re constantly working to eliminate bugs, so the app works the way you expect it to. In SoundSource 6.1, we’ve stamped out 33 distinct issues, including four (fortunately rare) crashes.

The most notable fix allows SoundSource to play nicely with audio playback from games running in MacOS’s “Game Mode”. Game on!

Get SoundSource 6.1 Today

SoundSource’s release notes always offer a comprehensive review of what’s changed, but why not simply try it yourself? If you’re already running SoundSource 6, visit the Settings window and click Check for Update to move up free of charge.

If you’re still running SoundSource 5, now’s a great time to upgrade. Download SoundSource 6, read about what’s new, and upgrade at a substantial discount.

And if you’re not yet using SoundSource at all? Head over to the SoundSource page to learn about it and download the free trial.

If you do anything with audio on your Mac, SoundSource 6.1 is ready to assist you.

The Aggregation of Marginal Gains

Improving our award-winning software 1% at a time.

If I had to identify a single principle that has allowed Rogue Amoeba to consistently develop high-quality, award-winning software for the last twenty-four years, it would be the aggregation of marginal gains.

You might know that term from the story of cycling coach Dave Brailsford. Brailsford took the helm of the British national team in 2003, after a century of underperformance. The team’s historic rankings were so poor, in fact, that at least one manufacturer refused to sell to the team. The vendor feared that the British using their bikes in competitions would sully the brand’s reputation.

How do you fix that? Previous coaches had attempted sweeping changes, but they had no success in improving things. By contrast, Brailsford approached the problem through what he called “the aggregation of marginal gains”. He and his team looked at every conceivable aspect of competitive cycling: seat comfort, rider posture, tread grip, rider self-care, and more. For each, they asked the same question: “How can we improve this by 1%?”.1

Over time, the team made hundreds of small improvements, and positive results followed. Five years after Brailsford took over, the British won 8 gold medals at the 2008 Olympics. Four years later, at the 2012 Olympics, they set multiple Olympic and world cycling records.2

When it comes to software, major updates with big new features catch attention, but we’re also constantly working on our own “1% improvements”. I’ve recently worked on a handful I’d like to highlight.

Customer Dashboard

Our Customer Dashboard provides a single location for our customers to access everything you might need from Rogue Amoeba, including a comprehensive record of your purchases:

This may seem like a larger initiative rather than a marginal improvement, but we’ve had automated systems in place for years that allowed users to recover license keys. Last year, we decided to improve upon that. First, we built a page that provided a list of your license keys, along with download links. Later, we provided the ability to manage the email address associated with purchases, then added purchase invoices. Our most recent enhancement adds automated notifications of eligible upgrades.

We started with something that matched our previous system in terms of functionality. From there, small, incremental changes rolled out features as they became available, resulting in a more robust system over time.

Permissions Window

Each year, MacOS includes more and more OS-level permissions. The resulting stream of prompts can interrupt work and cause friction, especially for folks trying out our software for the first time. While we can’t do much about this at the system level, in the spirit of incremental improvement, we asked ourself how we could make this marginally better for our users. The result is our centralized Permissions window, which has now been implemented across all of our apps:

This Permissions window provides a single interface to grant permissions to the app. It’s also a useful store of information, showing which permissions are required and which are optional, along with details on how each permission is used.

Unobtrusive Updates

Speaking of things that interrupt work, how many times have you launched software on your Mac only to be confronted with a workflow-interrupting update notification? We encourage our users to run our latest versions, but we also know that the moment you’re getting ready to launch a recording session is not the optimal time to update. So, we recently developed the unobtrusive update notifier to…um, unobtrusively notify users about updates, which they could then handle at their convenience:

This is a small improvement, and one that may well go unnoticed. But as Paul noted in the post announcing this change, this is a very nice quality-of-life improvement.

License Window

We provide free trials of all our software for users to evaluate, with a paid license required to unlock them for long term use. Over the years, we’ve made multiple refinements to our licensing system to improve that experience, including both visual and audible feedback, helpful messages to alert you to your eligibility for a discounted upgrade, and most recently, single-click-to-copy license codes:

We want users to have as little friction as possible, from evaluating the software with an easy-to-access free trial to unlocking the software once it’s been purchased.

Conclusion

In my 10 17 year tenure at Rogue Amoeba, I’ve seen the principle of incremental improvement applied consistently. Over time, small gains have aggregated to incredible functionality, accessibility, and performance for our users. While our software hasn’t won any Olympic medals, it has been used to facilitate Olympic broadcasts, among countless other uses. We’re proud of the steady stream of incremental improvements we’ve provided over the last twenty-four years, and look forward to continuing that practice long into the future.


Footnotes:

  1. You can read more via James Clear’s blog post. Clear is also the author of the book “Atomic Habits”, which focuses on getting 1% better every day. ↩︎

  2. Learn more about cycling at the 2008 Summer Olympics and records at the 2012 Summer Olympics. ↩︎

Free The Icons

Apple should end their prohibition on shapes in MacOS app icons

With last year’s release of MacOS 26 (Tahoe), Apple made a mess of app icons. In the first betas of MacOS 27 (Golden Gate), however, there are signs of a turnaround. We’re urging Apple to continue making improvements, by restoring the ability for MacOS app icons to have distinct shapes.

Apple’s Liquid Glass App Icons

In Tahoe, Apple modified the icons for dozens of their first-party apps to give them a “Liquid Glass” appearance. The changes were a substantial regression, leading to blurry, dumbed-down icons.

Two Automator icons, one pre-Tahoe with a detailed robot and one on Tahoe, with a hard-to-distiguish robot face.

With the recently unveiled Golden Gate, Apple has again updated their MacOS app icons. This time, however, the changes are genuine improvements. Here’s the refined Automator icon, for example:

The newer icon is sharper, with superfluous Liquid Glass removed. Dozens of Apple’s apps have seen similar updates. The result is that Golden Gate’s icons are superior to Tahoe’s, as this comparison from Basic Apple Guy shows. Seeing these improvements led me to think about another fix Apple should make in MacOS.

The Problem of Tahoe’s Dictated Squircles

With the Tahoe release, Apple didn’t just mess with their own icons. They also dictated the shape of every third-party app icon, forcing them to adopt the same prescribed squircle. Any icon that failed to do so found itself shrunk down and imprisoned in an ugly gray background, in order to fit Apple’s desired aesthetic.

Audio Hijack outside of, and inside, icon jail
Audio Hijack’s icon as it used to appear, and in Tahoe icon jail

To avoid this icon jail, developers were forced to redesign their icons to match Apple’s preferred form. After decades of beautiful, memorable Mac icons in varying shapes, Tahoe flattened personality to obtain bland uniformity. The platform is worse for it.

Past icons weren’t just more expressive. They were also more usable. Having distinct shapes provided a useful way to tell icons apart. Tahoe eliminates that cue by forcing everything into the same squircle, leaving color as the primary way to tell icons apart at a glance.

That falls down if you’ve got color vision deficiency, or even just multiple icons with similar color schemes.1 I’m looking at you, Slack and Photos. I have to look closely, because it’s so difficult to tell you apart now.

It Doesn’t Have to Be Like This

Apple’s prohibition on shapes is a step backward for both usability and creativity in app icons. Icons are now harder to distinguish because they’re no longer allowed to be distinctive. But there’s no technical reason for it. Apple could, and should, once again allow icons to take on a wide variety of shapes.

It’s clear that some people within Apple recognize that the transition to Liquid Glass introduced mistakes. They also appear to have the authority to fix those mistakes. Refinements to Apple’s own icons in Golden Gate are a welcome course correction, as is the much-celebrated Liquid Glass opacity slider. It’s time to correct the mistake of banning icon shapes as well.2

Apple should stop forcing every icon into the same squircle. Let’s return to a world of gorgeous app icons like these:

A collection of gorgeous old school icons

Free the icons.


Footnotes:

  1. With color now so critical to tell icons apart, it should be no surprise that the new “Clear” and “Tinted” icon styles added in Tahoe are seeing so little uptake. As Adam Engst noted, “[I]t’s nearly impossible to identify a particular app when they’re all clear or tinted squircles, as you can see below. My brain just shuts down when it sees them.”

    A sea of identical-looking icons on Tahoe, all tinted blue

    I’m not sure this “Tinted” style would be a good idea even if these icons had distinct shapes, but I know it’s a very bad one given their uniformity. ↩︎

  2. For folks within Apple, this was feedback filed as FB23388490 (“Third-Party App Icons Should Not Be Restricted to Apple’s Dictated Squircle Shape”). I imagine it is a duplicate many times over. ↩︎