Under The Microscope


Archive for February, 2017

JavaScript: Correctly Converting a Byte Array to a UTF-8 String

As you may know, the release of Airfoil for Mac 5.5 added support for streaming audio from your Mac to Google Chromecast devices. Despite the fact that Chromecast and AirPlay receivers perform very similar functions, their internal workings are very different, and making it all work together was no small feat. Streaming audio to the Chromecast actually required us to write our own custom receiver application to run on that device itself.

Chromecast applications are written in JavaScript, and while we’ve covered a lot of ground over the years, creating a network audio player in JavaScript was definitely a new challenge. While working on this, we encountered one particularly interesting problem which didn’t seem to have a ready-made solution. After much fruitless web searching, we ultimately solved this problem ourselves, and feel that solution is worth sharing with the world at large. Non-programmer users may wish to stop reading here (go check out Airfoil for Mac if you haven’t seen it yet!), but JavaScript coders, this is for you.

A JavaScript Problem

This issue we ran into was accurately rendering metadata strings — the information on what artist and song is currently playing. Airfoil converts these strings to a stream of bytes and sends them across the network to the Chromecast receiver, which converts them back to a string for display. This sounds simple enough, but there’s a catch: Many metadata strings require special handling.

An easy example of this is accented characters, as seen in band names ranging from Queensrÿche to Sigur Rós. When converting this text to a stream of bytes, the special characters need to be encoded with something like UTF-8. When converting from a stream of bytes back to a string, the UTF-8 must be decoded back the right special characters. This is a very old, very solved problem, but it is still surprisingly easy to screw it up.

Searching for “JavaScript convert bytes to string” will bring you to numerous code samples and Stack Overflow questions. Nearly all of these answers are wrong, or at least incomplete when it comes to correctly handling UTF-8 strings. The code samples generally look like this:

  function stringFromArray(data)
  {
    var count = data.length;
    var str = "";
    
    for(var index = 0; index < count; index += 1)
      str += String.fromCharCode(data[index]);
    
    return str;
  }

While this works fine for simple ASCII strings, it fails to correctly decode strings that contain special characters such as accented characters. The problem is that special characters get encoded into multi-byte sequences, but the simple loop calling fromCharCode() will treat every byte as if it were a single character. Special characters go in, and mangled sequences of ASCII come out.


Top: Mangled; Bottom: Correct

Our JavaScript Solution

Modern string handling means dealing with UTF-8 and correctly handling special characters, so this is a common problem that should have lots of proper solutions. Unfortunately, further searching for something like “JavaScript convert UTF-8 encoded bytes to string” currently provides no additional help, instead resulting in the same wrong results as the previous query.

Hopefully, we can change that. Here’s the solution for converting bytes to a string that we worked out, which correctly handles UTF-8 encoded special characters:

  function stringFromUTF8Array(data)
  {
    const extraByteMap = [ 1, 1, 1, 1, 2, 2, 3, 0 ];
    var count = data.length;
    var str = "";
    
    for (var index = 0;index < count;)
    {
      var ch = data[index++];
      if (ch & 0x80)
      {
        var extra = extraByteMap[(ch >> 3) & 0x07];
        if (!(ch & 0x40) || !extra || ((index + extra) > count))
          return null;
        
        ch = ch & (0x3F >> extra);
        for (;extra > 0;extra -= 1)
        {
          var chx = data[index++];
          if ((chx & 0xC0) != 0x80)
            return null;
          
          ch = (ch << 6) | (chx & 0x3F);
        }
      }
      
      str += String.fromCharCode(ch);
    }
    
    return str;
  }

As you can see, our version is longer than the commonly found, and incorrect examples. That additional code provides two big improvements. First, it will translate multi-byte UTF-8 sequences correctly, thereby preserving any special characters. As well, it will pick up on invalid UTF-8 sequences, so data corruption can be detected.

Hopefully, this solution will prove useful to future JavaScript coders! Remember, converting bytes to a string requires attention to special characters, and our more robust solution above should help.

SoundSource: The Sound Control That Should Be Built Into MacOS

Today, we’re pleased to unveil SoundSource, a tremendously handy new tool offering fast access to all your Mac’s critical sound settings!

From SoundSource’s menu bar icon, you can instantly configure the audio devices your Mac uses for Input, Output and Sound Effects. In seconds, you can adjust the volume for each of your audio devices or switch between connected devices. SoundSource can also enable the soft play-thru of audio from input devices. Use the Play-Thru window to monitor any connected input, such as a microphone, right through your headphones or other output.

SoundSource is a superior sound control in a tremendously convenient package. It tucks out of your way in the menu bar until you need it, then provides easy access to swap audio devices, adjust volumes, and more. It’s the sound control that should be built into MacOS, now available from Rogue Amoeba.1

Get It Now

Visit the SoundSource page to learn more and download our free trial for Mac OS X 10.10 (Yosemite) and higher. When you’re ready, you can purchase SoundSource right through our store.

Download Loopback Now

A Special Offer for Current Rogue Amoeba Customers

As a special bonus for our customers, folks who own a license for any of current MacOS applications are eligible to receive a complimentary license for SoundSource! If you already own one of our other products, click here to redeem this offer.

If you’ve been considering picking up Airfoil, Audio Hijack, or any of our other Mac apps, now’s a great time to do it. When you purchase, you’ll also earn a complimentary copy of SoundSource.


Footnotes:

  1. If the SoundSource name sounds familiar, you may have used our original SoundSource application back in the early 2000s. SoundSource 3 is a modern new version, offering the audio device controls from the original SoundSource, as well as the audio play-thru functionality once found in our LineIn application. Click here to read more. ↩︎

Making More Outside The App Store

Kapeli, makers of the developer tool Dash, recently posted about what happened when their app left Apple’s App Store. You may recall hearing about Dash due to the uproar that followed Apple’s decision to remove the well-loved app. Fortunately for Kapeli, the product was already set up for direct purchase, which meant they still had a way to sell it to customers. It turns out that in the first 100 days of Dash being distributed solely by Kapeli, it actually earned more revenue than when it was also sold through the App Store.


Image via https://blog.kapeli.com/100-days-without-the-app-store

There are undoubtedly multiple reasons for this result. Dash received a good deal of publicity as a result of Kapeli’s dispute with Apple, which likely brought in new customers. As well, it seems possible that professional developers are more willing than the average user to look outside the App Store for software. Still, it’s interesting to see how the App Store impacted Kapeli’s revenue.

With the exception of our audio editor Fission, all of Rogue Amoeba’s Mac apps are distributed and sold exclusively through our site. Popular tools like Audio Hijack and Airfoil provide concrete proof that products can thrive while never being sold through the Mac App Store. However, we have one application that followed a path similar to Dash’s. Our charmingly simple audio recording app Piezo was originally distributed in both the Mac App Store and via direct sales, but it has since left the App Store.

After seeing Kapeli’s chart, I was curious about the App Store’s impact on Piezo’s sales. The restrictions and limitations of the Mac App Store ultimately led us to remove Piezo on February 12th, 2016. We’ve now been selling it exclusively via our site for a year. This has provided about as perfect a real-world test case as one could hope for. Piezo’s removal came with minimal publicity, the price has remained constant at $19, and we’ve had no big updates or other major publicity for it in either 2015 or 2016.

So what do the numbers tell us? Here’s a chart showing unit sales for the last four quarters in which Piezo was sold through both the Mac App Store and directly via our website, as well as the subsequent four quarters when it was sold exclusively via direct sale.1

Piezo Units by Quarter

The Mac App Store previously made up about half of Piezo’s unit sales, so we might have expected to sell half as many copies after exiting the store. Instead, it seems that nearly all of those App Store sales shifted to direct sales. It appears that nearly everyone who would have purchased Piezo via the Mac App Store opted to purchase directly once that was the only option. Far from the Mac App Store helping drive sales to us, it appears we had instead been driving sales away from our own site, and into the Mac App Store.

While this chart doesn’t provide specific sales numbers, you can likely see that unit sales did drop slightly in 2016. Piezo’s removal from the Mac App Store does seem to have cost us a small number of sales. However, unit sales are far less important to a business than revenue. Let’s look at another chart, this time showing our revenue for the four quarters before removing Piezo from the Mac App Store, and the four quarters since.

Piezo Revenue by Quarter

In each of the four most recent quarters, Piezo brought in more revenue than it had in the corresponding quarter a year earlier. We earned more revenue when Piezo was available exclusively through our store than when we provided the App Store as another purchasing option.

This result might seem counterintuitive. Piezo’s price remained the same, and unit sales went down, so how could we have earned more revenue? The key to understanding this is remembering the cost of being in Apple’s App Stores — 30% off the top of every sale. Despite making slightly fewer sales, we earned more money by avoiding paying that oversized commission to Apple. Direct sales cost us just a few percent, so each direct sale of Piezo earns almost $5 more than a sale through the Mac App Store. As you can see, that really adds up.

Conclusion

I certainly won’t state that every developer will have this same success if they remove a product from the Mac App Store and distribute it exclusively through their own site. Your mileage will undoubtedly vary.

In our case, however, it’s clear that we were serving Apple, rather than Apple serving us. By removing Piezo from the Mac App Store, we stopped paying a commission to Apple for the many customers who had found Rogue Amoeba on their own. Better still, we were able to improve the quality of the product while simplifying our work considerably. Ultimately, that alone was enough to convince us that leaving the Mac App Store was the right move. The subsequent revenue increase we’ve seen is merely a nice bonus.


Footnotes:

  1. All eight quarters consist of exactly 91 days, to avoid 2016’s leap day skewing anything.↩︎

Our Software