Under The Microscope

Customer Service That Remembers

After a major upgrade, we contacted hundreds of folks directly to tell them their feature requests had been fulfilled.

An elephant wearing a headset

Back in December, we shipped a huge upgrade to our powerful audio control utility SoundSource. To spread the word, we announced SoundSource 6 on our blog and sent a mass mailing to our customer list. We also took a more targeted approach, contacting both existing and potential customers who’d previously reached out to us with specific requests. We went back to those folks to let them know about the changes that would interest them most.

This was a big effort, but it was well worth doing, and well worth talking about too.

Constant Groundwork

When you email our support team, we make sure you get a helpful reply, but our work doesn’t stop there. We also make sure to attach your report to relevant feature requests and bug reports in our internal issue tracker. Connecting the reports allows us to see how impactful specific changes could be, by providing a sense of how many people will be helped. Cross-linking issues and emails also enables us to get back to the exact message from someone who wanted a feature or told us about a bug, to glean further details.

It’s sort of like gardening. We do steady work every day, tending these requests. With major software updates, we’re presented with a harvest. And in SoundSource 6, we had dozens of features and bug fixes that became the fruit, allowing us to follow up with folks.

Building (and Re-Building) Systems

When I first joined back in 2010, Rogue Amoeba used an all-in-one tool called FogBugz for both handling support emails and filing bugs and features. As our needs changed (and FogBugz withered), we transitioned to two separate systems: Help Scout and GitHub. We handle customer support with Help Scout, and GitHub handles issue tracking and project management.

As we moved from an all-in-one solution to two distinct tools, we needed to build some plumbing to connect the two systems. We took that opportunity to improve the way feedback informs our development work. Thanks to some work with Help Scout’s custom fields, our support team can capture conversation-specific data that can then be referenced in GitHub, without bringing whole support exchanges into our product management tool.

Each time a new request comes in, we start by creating an issue in GitHub to establish the basic “what” of a feature, such as “Add AirPlay support to SoundSource”. Our tooling then makes it easy to pull customer quotes from conversations in Help Scout. The “why” gets established with real-world examples (“I want to apply audio effects to Spotify, then send its audio to my HomePods”) that offer helpful context, and allows specific user feedback to be heard. These quotes act as summaries, with the specific Help Scout conversations in question linked in case more details are desired.

Preparing Our Outreach

So, when it came time to promote SoundSource 6, we decided to write back with good news. To everyone. Though it would mean a lot of email-sending, we worked to develop a plan to personally let folks know of each feature or fix that was relevant to them.

To support this effort, we started a review of over 1,300 customer conversations linked to SoundSource feature requests, some dating as far back as 2019. We sorted these by feature type, which quickly showed what some of our most popular new features would be. For instance, we found nearly 250 folks who would benefit from the new Quick Configs and there were over 150 people for whom SoundSource’s powerful new AirPlay transmission would be a boon.

Then, we set out to write hundreds of emails to those individuals.

Crafting Emails

A real live person writes every email response our support team sends, and that human touch takes time. The process is helped along by text snippets, replies that we write once and then reuse and adapt to answer common questions.

This same Saved Replies functionality also helped us prepare the customer outreach for SoundSource 6. We started from a “fulfilled feature” saved reply to create those emails within Help Scout.

We used a fairly straightforward structure: a personalized greeting, an acknowledgement that some time might have passed since we were last in contact, and the announcement of the update. With some modifications, the resulting email was in place to send to those interested in SoundSource 6:

Hi {%customer.firstName,fallback=there%},

I’m emailing you from Rogue Amoeba regarding feedback you’d previously sent 
us (perhaps quite some time ago!) about SoundSource. We recently shipped a 
major upgrade, SoundSource 6, which you should check out.

[Feature-specific paragraph goes here.]

There’s much more you can read about on our “What’s New” page:

https://rogueamoeba.com/soundsource/whatsnew.php

SoundSource 6 costs $49, but if you’re moving from version 4 or 5, the
upgrade is discounted to just $25.

If you have any questions or more feedback to share, please don’t 
hesitate to reply.

The phrase “perhaps quite some time ago” did a lot of work, because some of our responses were in reply to years-old emails. However long it may have been, though, our systems ensured the user had not been forgotten by our team.

We could have sent one generic email that just listed new features, like “Added AirPlay streaming”, but it would have been a missed opportunity. Instead, we wrote 10 feature-specific variants that were much more helpful for the user, with the second paragraph customized to talk directly about what the user wanted. For instance, the AirPlay-focused email specifically mentioned HomePods and Sonos devices, along with per-app audio routing. This connected the current feature to the past request much more concretely.

We also included a nice bonus for these folks who had taken time to give us helpful feedback. As part of the initial sales push for SoundSource 6, we had created a limited-time coupon for new purchasers and provided it to our mailing list. So, we decided to include that discount in the email as well.

Human Review, Too

After we wrote our draft replies, and the technology to send messages was configured with Help Scout’s workflows, we still needed to confirm that each email actually should be sent. To do that, we again used a human touch.

We intentionally split our process into two steps, so we could validate each reply before it was sent:

  1. We would tag any conversation linked to an existing issue, to receive an initial review; and
  2. We would review each pending conversation, to confirm they should get the reply we developed.

As we worked through this, we reviewed batches of about 50 tabs at a time. To decide if an email should be sent to a customer, we checked Help Scout’s conversation history, as well as our own custom app for their order history. For example, if someone requested several features across different emails, we’d pick the one that was most impactful, then delete other drafts to avoid inundating them with multiple messages. For anyone who already upgraded to SoundSource 6, we could change the message to request feedback about how the new functionality was working for them.

This process was absolutely slower than one large bulk send. It was also deliberate and respectful of our customers’ inboxes.

A Gratifying Response

Our hope was that this outreach would land well with existing and potential customers. But would folks remember requests they’d made years ago – perhaps even before the pandemic?

They did. Our emails were well-received. Positive responses filled our inbox:

“That is some amazing helpdesk long-tail support flexing.”

“Such excellent work from you guys to link support tickets to development tickets and be able to do this. Congrats to the team.”

Many recipients upgraded, and were delighted with the new version. They also appreciated being remembered.

“The fact that you remembered my situation and took the time to personally alert me to the upgrade — that takes the cake.”

“I want to say, in all my years of post production work, which has included many feature requests, I’ve never had a vendor reach out to me to tell me that the request was added.”

One response, from a fellow support professional, particularly warmed my heart:

“While it’s nice and all — you writing me after five years telling me about it, that’s the kind of customer service I strive to offer myself. My hat is off.”

“We Heard You”

Contacting customers can drive new sales and develop goodwill. All too often, however, these points of contact feel impersonal or even robotic. While there’s certainly a place for mass communication (again, check out our mailing list – it’s great!), we wanted this outreach to be more personal.

Every day, customers reach out to explain their needs. That involves a certain amount of trust in the person on the other end, that they will actually listen and value their input. We strive for that kind of human connection, even in the face of so much technology that can strain it. Your email and your feedback are very important to us, and we want to demonstrate that with concrete actions – even years after the fact.

These messages, and the efforts behind them, can carry a simple mantra: “We heard you, we built it, and we hope you like it.”

Slow and Steady and Still Listening

Rogue Amoeba has been making Mac software since 2002, and we’ve always been in it for the long haul. I’ve been privileged to be part of the company for over half its life, but many of our customers have been with us even longer. Those relationships aren’t, and shouldn’t be, strictly transactional.

Software development can be slow, and features that seem simple might actually take years to implement properly. Priorities shift, and resources are limited, but customer requests don’t disappear. When they’re finally fulfilled, we want to share the good news with those it will matter most to. Personally following up on five-year-old feature requests isn’t efficient, but we’ve found that it’s been worth that extra effort. It’s certainly something we’re proud to be known for.

To everyone who contacted us over the years with ideas for SoundSource: thank you! Your feedback helped us shape what version 6 has become, and it continues to help guide us for the future. If you’ve asked about something we haven’t built yet, know that we haven’t forgotten.

We’re still listening, tracking, and planning. Please, as always, feel free to get in touch.

The elephant keeps walking

Audio Hijack 4.6 Is Now Available

It’s got MacOS 27 support, Transcribe updates, and more.

We’ve just shipped Audio Hijack 4.6, the latest version of our audio recording tool. This update includes several small bug fixes detailed in the comprehensive release notes, along with initial support for Apple’s forthcoming MacOS 27 (Golden Gate) release, with no known issues on the new OS.

In addition, we’ve spent some time on improvements to the built-in Transcribe block. With Transcribe, it’s long been easy to convert speech into text. This functionality has just been enhanced with a new transcription model that balances size, speed, and accuracy. The “Recommended” model gives results close to the large “High Accuracy” model, at speeds close to the “Low Resources” model. The Whisper.cpp backend that powers Transcribe has also been updated to version 1.9.2, which improves performance by reducing processor usage.

Audio Hijack 4.6 is a free update for all Audio Hijack 4 users. Just select “Check for Updates” from the “Audio Hijack” menu to download it.

If you haven’t used our flagship audio recorder before, give the Audio Hijack product page a visit to learn more and download the free trial.

The Sleep-Promoting Resilience of Boring Infrastructure

“Legacy” doesn’t have to be a bad word.

In last month’s post on incremental improvements, I highlighted some recent user-facing updates we’ve made here at Rogue Amoeba. Today, I’d like to share some details on the backend systems that have kept the company running strong for nearly a quarter of a century.

Measured against current development trends, Rogue Amoeba takes a very conservative approach to infrastructure. You might guess this is because we’re not a web app company, so we don’t have to think about infrastructure at scale. While there is a measure of truth to that, our infrastructure codebase is actually nearly double the size of the codebase for our commercial products. That infrastructure powers a wealth of systems, including our online store, Customer Dashboard, software updates, and Support Center.

Because of the importance of those systems, we’ve long avoided adopting untested technologies and complex architectures. A few weeks ago, while discussing the fact that one of our most recent rollouts found us “updating” to a 30-year-old technology, I jokingly asked our CTO Quentin, “What’s the opposite of being on the bleeding edge?” His response echoes one of the core tenets of our infrastructure philosophy:

“Being resilient.”

In our infrastructure planning, we regularly ask how we can make things less complicated, easier to maintain, and more resilient. When we make a change, ensuring our customers can access our site, download and purchase our software, and get the support they need is top of mind.

At a past job working in real-time transaction processing for a major fintech, I received the advice to deploy systems that won’t keep you up at night. That’s what we do at Rogue Amoeba. Most of our infrastructure decisions focus on decades-old, battle-tested technologies. Further, whenever we do update a dependency or migrate tech, those changes are rolled out slowly and deliberately.

And so, here are two recent examples of sleep-promoting systems we’ve deployed atop “boring” tech stacks:

FastCGI Migration

After we enhanced our software update system to show richer release notes, we noticed the changes were putting a relatively high strain on our server. That led us to take a hard look at some of the most legacy of our legacy systems, our CGI scripts. Over the years, we built these to handle things like update notifications, in-app support requests, and order fulfillment. With the Python foundation recently having removed the cgi library1, we decided it was time to start looking at other options.

There are many modern tech stacks we could have chosen to migrate these systems to, including AWS Lambda, Google Cloud Functions, and Cloudflare Workers. But instead, we asked, “What’s the simplest thing we can do to make these scripts significantly more performant?” That question was answered about 30 years ago, with the development of FastCGI.

After some initial testing and experimentation, we migrated all of our CGI scripts to FastCGI relatively easily. The performance improvement was notable, making it clear that “FastCGI” is named well:

This graph compares four performance metrics for one of our most heavily utilized APIs, across both the old CGI stack and the new FastCGI stack. “Mean” represents the average response time of all requests, and while that improved significantly, the worst-case scenarios are especially striking. To measure those, we used the common P90/P95/P99 SLA metrics.2, which say “90%/95%/99% of requests are handled faster than this.”

As you can see above, all three of those worst-case scenarios improved significantly under FastCGI. Could AWS Lambda have gotten us down to single-digit response times? Maybe. Would it have exponentially increased our infrastructure complexity and cost? Definitely. For us, that trade-off wasn’t worth it.

SQLite Migration

Another recent migration involved transitioning our primary order database from MySQL to SQLite. Folks who know about databases might think I wrote that backwards, but no.

Industry thinking around databases has shifted quite a bit over the years. By the early 2000s, MySQL and PostgreSQL had become the standard databases used in web development. That was followed by a brief surge of “NoSQL” advocacy that warned “you’d better be ready for big data.” As a result, many people migrated to systems like Mongo and Cassandra.

I have a good amount of experience with big data, having worked for both a major bioinformatics research institute that dealt with petabytes of genetic data3 and a fraud-analytics company that had to process billions of rows of transactional data. While database engines like Mongo and Cassandra have their place, the majority of people who adopted those systems are wielding way more database power than they need. A data engineer friend of mine once remarked that powering most websites with Cassandra or Mongo is like buying a fully loaded SUV to drive to the corner store.

While MySQL and PostgreSQL were dominating the web application world, around the same time period a smaller, lesser-known database system called SQLite had begun taking over the embedded systems space. You’d probably be surprised by the number of things in your life that are powered by SQLite databases. It is now the most widely used database system in the world4, powering cell phones, automotive systems, web browsers, modern planes, and even satellites.5

Unlike larger database engines that require dedicated server setups and separate permission sets, SQLite databases consist of a single file that can sit alongside the rest of your web application code. Despite their simplicity, on modern hardware using WAL6, SQLite can easily handle tens of thousands of reads and writes per second. On more sophisticated hardware with software-level optimizations, it can go orders of magnitude beyond that.7 Since SQLite is a file-system-based DB, permissions are a breeze to manage, and developments like LiteFS make replication just as straightforward.

The best part is you can back up and restore your entire infrastructure and data stack with a simple file copy, or use tools like Litestream to stream changes into object storage.

SQLite databases already powered most of our internal systems at Rogue Amoeba, but the order database was a holdout, still relying on MySQL. We knew migrating it would be a long-term win, but the migration involved extensive code changes. Further, since these systems handle order fulfillment, the method and timing of the migration were critical as we didn’t want to deploy multiple sweeping changes simultaneously. We moved slowly, rolling out the necessary changes in heavily tested phases. Eventually, we ran both database systems in parallel for a time, until we were satisfied that the new system performed well.

The final result has exceeded our expectations. Performance is spectacular and DB operations like schema changes now complete in milliseconds, compared to the sometimes hours it took with MySQL. Best of all, we have one fewer set of servers, permissions, and network configs to manage.

Conclusion

The term “legacy code” is often used in a negative light, as something that needs to be replaced simply because of its age. But with so many recent stories about AI-deleted codebases, poisoned NPM dependency graphs, the left-pad fiasco, and cloud outages that take down half the internet, I prefer the other definition of “legacy”: a long-lasting impact of events and actions.

Our infrastructure systems are not exciting, and we’re just fine with that. Keeping our infrastructure “boring” and occasionally embracing battle-tested, “legacy” technologies allows us to keep our focus on our true legacy: building award-winning software that improves the lives of our customers.


Footnotes:

  1. See PEP 594 for a discussion on the rationale for removing cgi support from the Python standard library. ↩︎

  2. SLAs, or “Service-level agreements”, are the set of performance and uptime promises service providers guarantee to customers. P90, P95, and P99 thresholds are often part of those agreements. ↩︎

  3. If you need to handle really big data, take a look at that research facility’s ITS infrastructure blog for some tips. ↩︎

  4. See SQLite’s deployment stats page for details. ↩︎

  5. For more fun examples, click here and here. ↩︎

  6. WAL stands for “Write-Ahead Logging”, which allows SQLite to append transactions to a separate log file, then commit them in batches. It’s not only faster, it also prevents readers and writers from blocking one another. For more details, see https://sqlite.org/wal.html. I suppose I should note that WAL mode adds two more files, alongside the main database file. ↩︎

  7. Here’s an impressive example of someone reaching 4 Million queries per second with SQLite. ↩︎

Fifteen Fantastic Years

Our 15 year club has its second member.

It’s once again time to celebrate an employee milestone here at Rogue Amoeba. Membership in our 15 year club recently doubled, with Grant Farr joining that exclusive group. It is an honor and a privilege to celebrate him today.

Helming Audio Hijack

When we marked Grant’s five and ten year milestones, the primary focus was on Audio Hijack, and with good reason. From 2011 through today, his steady hand has guided our flagship to continued excellence. In 2015, we shipped Audio Hijack 3, the app’s revolutionary leap forward after over a decade of version 2. In 2022, Audio Hijack 4 shipped, an evolutionary update. In between, and since, we’ve had point updates that added new features and functionality.

Grant led the app through all of that, and he continues to lead it today. His work has made Audio Hijack remarkably stable, even as the app has evolved and changed, with year after year of steady improvements really adding up.

Audio Hijack is now up to version 4.5, and it remains one of the Mac’s most powerful and popular tools. Of course, there are more updates and improvements in the works too, and we know you’ll love what Grant’s been cooking up.

And Piezo

A decade and a half of leading Audio Hijack would be plenty, but that’s not the only audio recording tool Grant has worked on. Since 2011, he’s also served as the lead developer on Piezo, our charmingly simple recording app. Piezo was originally created to make recording as easy as possible, and nearly 15 years later, it’s still doing that. That’s possible in no small part thanks to Grant’s work, which created a dependable app that doesn’t require a lot of upkeep.

And Farrago 1

We can’t forget the third app Grant’s led here at Rogue Amoeba: Farrago. What began as an idea to include inside of Audio Hijack grew into the Mac’s best soundboard app. Development of Farrago was eventually handed off for version 2, but the app is still built on top of the wonderful work Grant did to create Farrago 1.

Even More

Grant’s contributions have been crucial to even more of our product lineup. Much of Grant’s user interface work is actually shared across multiple applications. Big parts of work he’s done have then been re-used in other apps, including Loopback and SoundSource. In ways both big and small, Grant’s work helps countless users every single day.

Take Your Pick

With Chris Barajas’s anniversary last year, we decided new members of Rogue Amoeba’s 15 year club would receive an original Apple watch:

The original Apple watch, close up

At the time, I noted that the timepiece in question dated to 1995 and Mac System 7.5. Long-time Apple fans might recall that this watch was specifically part of a promotion to encourage users to purchase an OS upgrade for their Macs:

An ad from 1995 offering the choice between two free gifts, a watch or Conflict Catcher 3, when purchasing System 7.5 for their Mac
[Image credit: Hodinkee]

We decided we’d one-up Apple’s promotional offer by providing both promo options. Yes, in addition to a beautiful and bizarre original Apple watch, folks who work at Rogue Amoeba for 15 years also earn a copy of Conflict Catcher 3:

Grant will need to locate his own floppy drive, however.

Cheers to Grant

For over 15 years, Grant has been writing rock-solid code for Rogue Amoeba. Thanks to him, Audio Hijack has been incredibly stable, and that’s crucial for a production recording tool. His work also led to the low-maintenance delight that is Piezo, gave a strong foundation to Farrago, and provided crucial parts of Loopback and SoundSource. Not bad!

The positive impact Grant has had on Rogue Amoeba can’t be overstated, and we’re so glad to have him on our team. Thank you for all you’ve done, Grant, and here’s to many more years working together!

SoundSource 6.1 Is Here

You’ll find dozens of improvements in this free update.

We’ve just released SoundSource 6.1, a free update that brings dozens of refinements and improvements to all SoundSource 6 users. Read on for more details, or just click to download it immediately.

Continued Progress

Last winter, we shipped SoundSource 6. It was a major upgrade to our popular audio control utility that brought a bevy of new features. Today’s update builds on that progress, with a few small new features, some big refinements and optimizations, and more than two dozen bug fixes.

More Ways to Access Quick Configs

SoundSource’s quick configs save complete audio configurations for your Mac, then allow you to switch between them instantly. We’ve just added two new ways to select your quick configs.

Most notably, you can now put your quick configs right in your menu bar, as seen to the right. Enable the optional menu bar icon with the switch found in the Manage Quick Configs sheet. As well, SoundSource’s alternate menu (accessed by Control-clicking the main menu bar icon) now has a Quick Configs sub-menu.

Access your quick configs, quicker than ever.

Clearer Volume Levels in the Menu Bar

This update brings another notable change to the menu bar, as we’ve improved the volume level indicators found there. The previous three dots in SoundSource’s menu bar icon have now been replaced with four wider levels.

This updated art is shown when SoundSource’s Menu Bar Icon (in the Appearance settings) is set to either the Speaker (SoundSource) or Match Output Device.

Refinements for Bluetooth, Audio Devices, and Audio Units

Bluetooth outputs can often cause headaches, but this update reduces the pain. We’ve eliminated several issues that could cause audio glitches and dropouts when playing to your Bluetooth devices.

Speaking of audio hardware, SoundSource’s Audio Devices window has also been updated. It now includes “Preferred Device Order” controls for MacOS’s “Sound Effects” output, in addition to ensuring that devices you mark as hidden correctly stay hidden everywhere.

If you use SoundSource to enhance audio on your Mac, we’ve got plenty for you as well. We made multiple improvements to Audio Units in version 6.1. Of particular note, newly installed effects are now instantly ready for use in SoundSource.

Substantial Optimizations

This update also fine-tunes the ARK plugin that powers audio handling, making it more efficient than ever. To minimize resource consumption, we worked around several issues in MacOS itself. Recent tests show CPU usage down 18-35%.

We‘ve also elevated the experience when using SoundSource to send to AirPlay devices. Optimizations in streaming code allow for improved synchronization between AirPlay outputs, along with better connection reliability.

Bugs, Squashed

Bugs be goneWe’re constantly working to eliminate bugs, so the app works the way you expect it to. In SoundSource 6.1, we’ve stamped out 33 distinct issues, including four (fortunately rare) crashes.

The most notable fix allows SoundSource to play nicely with audio playback from games running in MacOS’s “Game Mode”. Game on!

Get SoundSource 6.1 Today

SoundSource’s release notes always offer a comprehensive review of what’s changed, but why not simply try it yourself? If you’re already running SoundSource 6, visit the Settings window and click Check for Update to move up free of charge.

If you’re still running SoundSource 5, now’s a great time to upgrade. Download SoundSource 6, read about what’s new, and upgrade at a substantial discount.

And if you’re not yet using SoundSource at all? Head over to the SoundSource page to learn about it and download the free trial.

If you do anything with audio on your Mac, SoundSource 6.1 is ready to assist you.