Airfoil
Audio Hijack
Farrago
Fission
Loopback
Piezo
Soundsource
Your Screen. On A Screen. On Screen!
Posted By Guy English on February 21st, 2008
I’m a real sucker for cute and slick little tricks, those small things that seem pretty cool when you see them and leave you thinking, “How’d they do that?”. For instance, if you’ve used an iPhone, you’ve no doubt seen that it bounces when it hits the bottom of a list. It doesn’t have to do this, but it’s certainly a cool effect. It’s also more natural, which is always a positive thing. This week, I was happy to be able to help add a similarly-cool trick to the newest version of Airfoil.
Perhaps the most interesting new feature of the the “Apple TV Take 2” update, as far as Rogue Amoeba is concerned, is its ability to receive streamed audio via iTunes. Because the Apple TV is an AirTunes client, you can now use Airfoil to send any audio to it. It’s in sending to the Apple TV that I was able to inject a bit more spice into Airfoil.
What we did
When you use iTunes to send audio to the Apple TV over AirTunes, iTunes sends metadata as well, causing the Apple TV to display the track name, the artist, the album, and album artwork. Once we had Airfoil properly talking to the Apple TV, we turned to working out what exactly we’d send for metadata.
In Leopard, Apple has shipped beautiful icons that represent all the recent models of Mac capable of running Mac OS X 10.5. From Mac Pros and Minis to the new MacBook Air, there’s an icon for each sitting on your system. You can see a slightly scaled-down version of their iMac icon to the right. We actually make use of these icons Airfoil already – when an Airfoil Speakers client shows up in Airfoil’s list, we show the associated artwork to help identify the client machine. However, these images are tiny, just 16×16, so you might not even notice that they’re customized for your machine. With Apple TV, however, we thought it’d be a nice touch to display a full-sized 512×512 image of the Mac that’s sending the audio.
Mike set this up easily enough and it certainly looked nice. On Tiger, however, we could only use a generic icon of an Apple Display. That set Paul to thinking about showing something on that display to help identify the machine. He hit on the idea of showing the desktop of the machine, superimposed on the icon. To quote him, “It’d only be mildly useful, but it would be sexy, and it might be easy?”.
I agreed that this was a sexy idea, but would it be easy? There was only one way to find out. Ultimately, it took some fancy footwork and a little bit of craziness on my part, but I was able to make this happen in just a few hours. Along the way, this actually morphed into a Leopard-only feature. So when you’re streaming to an Apple TV via Airfoil running on Leopard, we send along the image of your Mac and overlay your desktop on it. Put simply, it’s damn cool to see. You can see a sample below, – we’re generating this image live, as we’re sending iTunes audio to the Apple TV via Airfoil running on a MacBook Air1. It’s really best experienced in person though, so if you’ve got a Leopard mac and an Apple TV, stop reading and grab Airfoil right now to check it out.
“How’d they do that?”
So, how did we make this happen? To start, we ask the system for the image of the Mac model we’re running on (using [NSImage imageNamed: NSImageNameComputer]). That gives us one of those giant 512×512 pixel images representing the local machine. Then we need to figure out where on that image the “screen area” is – the blue desktop that we can overwrite. We debated hard-coding that information for all the models that were there, but that’s not terribly elegant, as it would fail any time Apple brings out a new Mac or if they changed the included images even slightly.
Instead, I decided we could do this with some image processing magic, finding the part of the image that shows the blue Mac OS X default desktop image. By finding colors between certain hue, brightness and saturation ranges, it’s possible to work out which part of the image represents the Mac’s screen. If a screen isn’t found, as in the cases of a G5 tower or Mac mini, then an image of a nice Apple Cinema Display is substituted in. The Windows Blue Screen Of Death beige monitor seen to the left was also under consideration, but the Cinema Display is just a tiny bit nicer.
At this point, we’ve got an image of a Mac or an Apple display and we know where the “desktop” area is on that image. That means it’s time to get the new desktop image to be superimposed over this. Originally we thought of doing a simple screen shot, but this raised security concerns. Mike envisioned, quote, “some upset user coming to us because he forgot to close his porno browser before he started playing music on his 52″ plasma screen”. A simple “Sorry Mom!” probably doesn’t fix the trauma there.
With a bit more discussion, we settled on the “Apple Product Shot” style – just the Desktop, Dock and Menubar. This provides a nice, clean look but it’s still obviously recognizable as your machine. Using the great new CGWindow API in Leopard, I was able to get just that without much difficulty. We also decided to add in the source application from which Airfoil is pulling audio. Here are a couple more examples, with Pandora from a Mac mini (so we substituted the generic Apple display) and Front Row from a MacBook Pro.
So that’s “how we did that”, and the final results are really pretty exciting. When you use Airfoil to send audio to your AppleTV, you’ll be treated to a great-looking rendition of your Mac, replete with its desktop and source application visible. It’s like magic! Don’t take my word for it though – you can get Airfoil 3.1 and see for yourself!
1. Why would we send iTunes via Airfoil? For use with Airfoil Speakers, of course. iTunes doesn’t acknowledge Airfoil Speakers as it can’t authenticate with it. Instead, we get Airfoil to pull audio from iTunes. Then we can send audio to the Apple TV, along with Airfoil Speakers on Mac or PC and of course, AirPort Express units. ↩
Mike Abdullah says:
February 21st, 2008 at 11:43 amWhat can I say, that is some amazingly cool technology there.
Patrick says:
February 21st, 2008 at 12:03 pmProgrammers are gods.
Brian says:
February 21st, 2008 at 12:38 pmSo, can I send Garageband output to the stereo using my airport express (or my Apple TV once I finally break down and get an HDTV)?
This sounds great. The fact that you can only use Airport express with iTunes is a real bummer sometimes. One of those things where you just KNOW that it’s entirely possible to do what you want, but you can’t (and one of those things your lame PC using friends like to bring up).
ScottGuelich says:
February 21st, 2008 at 1:02 pmVery cool!
On a related note, have you noticed that although the CGWindow functions are public and even demonstrated in the SonOfGrab sample code, they aren’t included in any of the doc sets? I’m using them in some code, but I have this small nagging worry that Apple might decide they shouldn’t have been public after all. (filed radr 5757244 to get them fully documented)
Mike says:
February 21st, 2008 at 1:29 pmBrian: That’s pretty much the idea behind Airfoil, sounds like it’s just what you need. Give the trial a shot and see how you like it.
ScottGuelich: When you’ve worked with CoreAudio as much as we have, APIs with no documentation outside of the headers no longer cause any worry. It often takes Apple a while to put out documentation for these things, especially obscure Carbon-y things like CGWindow. I don’t imagine there’s any risk of it going away, although filing a bug to get real docs is a good move.
Paul says:
February 24th, 2008 at 2:37 amMike Abdullah, Patrick: Ain’t it the truth?
Matt P says:
February 25th, 2008 at 2:52 pmCorrect me if I’m wrong, but shouldn’t Airfoil just send the metadata if it’s streaming iTunes? Is this not possible, or is this a decision that was made?
Grover says:
February 25th, 2008 at 3:24 pmIs there any way to make use of this incredibly awesome feature if you’re using a Mac Mini with Front Row as a HTPC? That is, a way that doesn’t require me to exit front row and double-click on the speaker application?
Blueberry says:
February 25th, 2008 at 3:52 pmMatt P: Metadata from iTunes is possible, but the trouble is that AirFoil is for streaming audio from any application not just iTunes, and so if they did that they would need to write code to extract meta data from each and every application that could conceivably be used with it for such a thing to work well, and they’d have to maintain that in to the future too. Very messy sounding.
Tres says:
February 25th, 2008 at 4:26 pmBrilliant. Yet another reason I will be purchasing Airfoil as soon as I get my Apple TV. I do have to ask, though, if you can grab the “live” desktop and send it through Airfoil, could you also add an option to send the current album art (if using iTunes as the source)? Great work, guys.
Paul says:
February 25th, 2008 at 5:28 pmMatt P., Tres: As noted on in a comment another post, we could grab metadata, but don’t currently. As Blueberry correctly notes, getting that metadata is a bit messy, as there are many apps in use with Airfoil. For the 3.1, we settled on a standard image and text to send. For the future, we’ll consider slurping from other apps.
Grover: Front Row on the Mac Mini has no capacity for receiving audio, I’m afraid. You can do it with Airfoil Speakers though, and once launched, you wouldn’t need to drop out of Front Row. Open up Airfoil Speakers, and go to the Preferences. Turn on the Start at Login pref and turn off the Float pref, then close the window.
Now, Airfoil Speakers will always be running (unless you expressly quit it), and the Mac Mini will always be ready to receive audio. It will not receive any visuals, though this is something that could change in a future Airfoil Speakers update.
FuturDreamz says:
February 25th, 2008 at 10:10 pmI’ve set my desktop to rotate through my collection every minute. will it acknowledge that change?
Rob L. says:
February 25th, 2008 at 11:31 pmSo, about this:
Pardon me if it’s a dumb question, but where on the system might I find these beautiful icons? I might just want to change my hard drive icon to the appropriate one, if I could only locate it :)
KenJohnson says:
February 26th, 2008 at 12:56 am/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources
As it’s a bundle, you’ll need to Ctrl click and select View Package Contents.
Nick C says:
February 26th, 2008 at 4:57 amThe main thing that frustrates me with using AppleTV to stream from iTunes is that if I am playing an internet radio station, like Radio Paradise, that pushes track info out as part of its stream, the AppleTV screen does not show this information. So I have to go to my Mac in another room to find out what great track it is that is playing, which should be unnecessary. I take the point about every application being different, but surely a special-case for iTunes would be worthwhile? Apart from anything else, it would add value to using Airfoil to stream from iTunes rather than simply using the built-in AirTunes functionality, which doesn’t provide track data from internet radio streams.
Incidentally I believe this is because Apple is not keen on people using AppleTV for internet radio because it is a way of getting unlimited content for free. Otherwise I can’t really understand why they haven’t supported internet radio better, or better still implemented the Internet Radio tuner feature from iTunes on AppleTV. Personally I think this is a shame, as additional functionality of this type would surely help to sell AppleTV, and most people listen to radio when they are doing something else so not in a position to watch purchasable video anyway. /rant
Arthur says:
February 26th, 2008 at 6:49 amNeat, but I’m not convinced that replacing a screenless icon improves usability. I have a mini and an iMac and it’s much easier to identify them from their icons rather than the desktops.
Paul says:
February 26th, 2008 at 12:01 pmFuturDreamz: Airfoil pulls the screen when it first connects, and updates it when you change sources. We’re not updating at any other time, because sending updates causes the Apple TV display to whoosh out in a rather distracting way, but any time it connects, it should get the latest desktop.
Rob L.: KenJohnson has the location. There are lots of nice icon files in there, but you’re looking for com.apple.* files.
Nick C: To be sure, Apple themselves should send info when sending a radio stream with iTunes. They also should kill the progress bar, but they don’t, yet anyway. As far as a special case for iTunes goes, yes and no. Yes, iTunes is popular, but you don’t need Airfoil if you’re using iTunes (unless you’re using Airfoil Speakers too). Anyhow, as I noted, this is 3.1, it’s not “the last version of Airfoil ever”. We’ll see what happens in the future.
As for not supporting internet radio, it’s also a matter of priorities. Internet radio isn’t going to be as popular as TV and movies on this device. Hopefully they’ll get to improving it, but don’t read too much into how things are on day 1 (or even day 1, Take Two).
Arthur: We actually went back and forth on this – the desktop definitely looked nicer than the static iMac, Mini, or tower, and personally, I recognize machines from their desktop backgrounds. We had to make a call, and we did – ultimately, it’s not really something worth too much debate 8).
Tom says:
February 26th, 2008 at 2:45 pm“…but the Cinema Display is just a tiny bit nicer.”
Yes.
Just a tiny bit.