An iPhone Core Audio brain dump

转自：http://www.subfurther.com/blog/2009/04/28/an-iphone-core-audio-brain-dump/

Twitter user blackbirdmobile just wondered aloud when the Core Audio stuff I’ve been writing about is going to come out. I have no idea, as the client has been commissioning a lot of work from a lot of iPhone/Mac writers I know, but has a lengthy review/rewrite process.

Right now, I’ve moved on to writing some beginner stuff for my next book, and will be switching from that to iPhone 3.0 material for the first book later today. And my next article is going to be on OpenAL. My next chance for some CA comes whenever I get time to work on some App Store stuff I’ve got planned.

So, while the material is still a little fresh, I’m going to post a stream-of-consciousness brain-dump of stuff that I learned along the way or found important to know in the course of working on this stuff.

• It’s hard. Jens Alfke put it thusly:
  

  “Easy” and “CoreAudio” can’t be used in the same sentence.  CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation.

• That said, tweets like this one piss me off. Media is intrinsically hard, and the typical way to make it easy is to throw out functionality, until you’re left with a play method and not much else.

• And if that’s all you want, please go use the HTML5 <video> and<audio> tags (hey, I do).

• Media is hard because you’re dealing with issues of hardware I/O, real-time, threading, performance, and a pretty dense body of theory, all at the same time. Webapps are trite by comparison.

• On the iPhone, Core Audio has three levels of opt-in for playback and recording, given your needs, listed here in increasing order of complexity/difficulty:

  1. AVAudioPlayer – File-based playback of DRM-free audio in Apple-supported codecs. Cocoa classes, called with Obj-C. iPhone 3.0 adds AVAudioRecorder (wasn’t sure if this was NDA, but it’s on the WWDC marketing page).
  2. Audio Queues – C-based API for buffered recording and playback of audio. Since you supply the samples, would work for a net radio player, and for your own formats and/or DRM/encryption schemes (decrypt in memory before handing off to the queue). Inherent latency due to the use of buffers.
  3. Audio Units – Low-level C-based API. Very low latency, as little as 29 milliseconds. Mixing, effects, near-direct access to input and output hardware.

• Other important Core API’s not directly tied to playback and recording: Audio Session Services (for communicating your app’s audio needs to the system and defining interaction with things like background iPod player, ring/silent switch) as well as getting audio H/W metadata, Audio File Services for reading/writing files,Audio File Stream Services for dealing with audio data in a network stream, Audio Conversion Services for converting between PCM and compressed formats (and vice versa), Extended Audio File Services for combining file and conversion Services (e.g., given PCM, write out to a compressed AAC file).

• You don’t get AVAudioPlayer or AVAudioRecorder on the Mac because you don’t need them: you already have QuickTime, and the QTKit API.

• The Audio Queue Services Programming Guide is sufficient to get you started with Audio Queues, though it is unfortunate that its code excerpts are not pulled together into a complete, runnable Xcode project.

• Lucky for you, I wrote one for the Streaming Audio chapter of thePrags’ iPhone book. Feel free to download the book’s example code. But do so quickly — the Streaming Audio chapter will probably go away in the 3.0 rewrite, as AVAudioRecorderobviates the need for most people to go down to the Audio Queue level. We may find some way to repurpose this content, but I’m not sure what form that will take. Also, I think there’s still a bug in the download where it can record with impunity, but can only play back once.

• The Audio Unit Programming Guide is required reading for using Audio Units, though you have to filter out the stuff related to writing your own AUs with the C++ API and testing their Mac GUIs.

• Get comfortable with pointers, the address-of operator (&), and maybe even malloc.

• You are going to fill out a lot of AudioStreamBasicDescriptionstructures. It drives some people a little batty.

• Always clear out your ASBDs, like this:

  
  memset (&myASBD, 0, sizeof (myASBD))
  

  This zeros out any fields that you haven’t set, which is important if you send an incomplete ASBD to a queue, audio file, or other object to have it filled in.

• Use the “canonical” format — 16-bit integer PCM — between your audio units. It works, and is far easier than trying to dick around bit-shifting 8.24 fixed point (the other canonical format).

• Audio Units achieve most of their functionality through setting properties. To set up a software renderer to provide a unit with samples, you don’t call some sort of a setRenderer() method, you set the kAudioUnitProperty_SetRenderCallback property on the unit, providing a AURenderCallbackStruct struct as the property value.

• Setting a property on an audio unit requires declaring the “scope” that the property applies to. Input scope is audio coming into the AU, output is going out of the unit, and global is for properties that affect the whole unit. So, if you set the stream format property on an AU’s input scope, you’re describing what you will supply to the AU.

• Audio Units also have “elements”, which may be more usefully thought of as “buses” (at least if you’ve ever used pro audio equipment, or mixing software that borrows its terminology). Think of a mixer unit: it has multiple (perhaps infinitely many) input buses, and one output bus. A splitter unit does the opposite: it takes one input bus and splits it into multiple output buses.

• Don’t confuse buses with channels (ie, mono, stereo, etc.). Your ASBD describes how many channels you’re working with, and you set the input or output ASBD for a given scope-and-bus pair with the stream description property.

• Make the RemoteIO unit your friend. This is the AU that talks to both input and output hardware. Its use of buses is atypical and potentially confusing. Enjoy the ASCII art:

  
                           -------------------------
                           | i                   o |
  -- BUS 1 -- from mic --> | n    REMOTE I/O     u | -- BUS 1 -- to app -->
                           | p      AUDIO        t |
  -- BUS 0 -- from app --> | u       UNIT        p | -- BUS 0 -- to speaker -->
                           | t                   u |
                           |                     t |
                           -------------------------
  

  Ergo, the stream properties for this unit are

  	Bus 0	Bus 1
  Input Scope:	Set ASBD to indicate what you’re providing for play-out	Get ASBD to inspect audio format being received from H/W
  Output Scope:	Get ASBD to inspect audio format being sent to H/W	Set ASBD to indicate what format you want your units to receive

• That said, setting up the callbacks for providing samples to or getting them from a unit take global scope, as their purpose is implicit from the property names:kAudioOutputUnitProperty_SetInputCallback andkAudioUnitProperty_SetRenderCallback.

• Michael Tyson wrote a vital blog on recording with RemoteIO that is required reading if you want to set callbacks directly on RemoteIO.

• Apple’s aurioTouch example also shows off audio input, but is much harder to read because of its ambition (it shows an oscilliscope-type view of the sampled audio, and optionally performs FFT to find common frequencies), and because it is written with Objective-C++, mixing C, C++, and Objective-C idioms.

• Don’t screw around in a render callback. I had correct code that didn’t work because it also had NSLogs, which were sufficiently expensive that I missed the real-time thread’s deadlines. When I commented out the NSLog, the audio started playing. If you don’t know what’s going on, set a breakpoint and use the debugger.

• Apple has a convention of providing a “user data” or “client” object to callbacks. You set this object when you setup the callback, and its parameter type for the callback function isvoid*, which you’ll have to cast back to whatever type your user data object is. If you’re using Cocoa, you can just use a Cocoa object: in simple code, I’ll have a view controller set the user data object as self, then cast back to MyViewController* on the first line of the callback. That’s OK for audio queues, but the overhead of Obj-C message dispatch is fairly high, so with Audio Units, I’ve started using plain C structs.

• Always set up your audio session stuff. For recording, you must use kAudioSessionCategory_PlayAndRecord and callAudioSessionSetActive(true) to get the mic turned on for you. You should probably also look at the properties to see if audio input is even available: it’s always available on the iPhone, never on the first-gen touch, and may or may not be on the second-gen touch.

• If you are doing anything more sophisticated than connecting a single callback to RemoteIO, you may want to use an AUGraph to manage your unit connections, rather than setting up everything with properties.

• When creating AUs directly, you set up aAudioComponentDescription and use the audio component manager to get the AUs. With an AUGraph, you hand the description to AUGraphAddNode to get back the pointer to anAUNode. You can get the Audio Unit wrapped by this node withAUGraphNodeInfo if you need to set some properties on it.

• Get used to providing pointers as parameters and having them filled in by function calls:

  
  AudioUnit remoteIOUnit;
  setupErr = AUGraphNodeInfo(auGraph, remoteIONode, NULL, &remoteIOUnit);
  

  Notice how the return value is an error code, not the unit you’re looking for, which instead comes back in the fourth parameter. We send the address of the remoteIOUnit local variable, and the function populates it.

• Also notice the convention for parameter names in Apple’s functions. inSomething is input to the function, outSomethingis output, and ioSomething does both. The latter two take pointers, naturally.

• In an AUGraph, you connect nodes with a simple one-line call:

  
  setupErr = AUGraphConnectNodeInput(auGraph, mixerNode, 0, remoteIONode, 0);
  

  This connects the output of the mixer node’s only bus (0) to the input of RemoteIO’s bus 0, which goes through RemoteIO and out to hardware.

• AUGraphs make it really easy to work with the mic input: create a RemoteIO node and connect its bus 1 to some other node.

• RemoteIO does not have a gain or volume property. The mixer unit has volume properties on all input buses and its output bus (0). Therefore, setting the mixer’s output volume property could be a de facto volume control, if it’s the last thing before RemoteIO. And it’s somewhat more appealing than manually multiplying all your samples by a volume factor.

• The mixer unit adds amplitudes. So if you have two sources that can hit maximum amplitude, and you mix them, you’re definitely going to clip.

• If you want to do both input and output, note that you can’t have two RemoteIO nodes in a graph. Once you’ve created one, just make multiple connections with it. The same node will be at the front and end of the graph in your mental model or on your diagram, but it’s OK, because the captured audio comes in on bus 1, and some point, you’ll connect that to a different bus (maybe as you pass through a mixer unit), eventually getting the audio to RemoteIO’s bus 0 input, which will go out to headphones or speakers on bus 0.

I didn’t come up with much (any?) of this myself. It’s all about good references. Here’s what you should add to your bookmarks (orTogether, where I throw any Core Audio pages I find useful):

• coreaudio-api mailing list — lots of useful stuff in the archives, and very helpful Apple engineers (provided you’ve done your homework and aren’t asking questions with obvious, readily-available answers).
• Core Audio Overview
• Audio Unit Programming Guide
• Audio Session Programming Guide
• OS X CoreAudio Host Tutorial — negative example, don’t read this. I made a bunch of mistakes based on this, even after translating Mac to iPhone (e.g., RemoteIO unit instead of AUHAL). Biggest one: calling AUGraphOpen(AudioGraph) after setting up nodes. This is bad because AUGraphNodeInfo doesn’t work unless the graph is open. Initialize late, but open early.
• Using a RemoteIO AudioUnit for low-latency recording — Michael Tyson initially posted his RemoteIO stuff in this Apple devforums thread, and there’s a long follow-on discussion with a lot of developers building on that basis.
• Technical Note TN2091: Device input using the HAL Output Audio Unit