Sound programming notes

Terminology

• bufferSize is now usually[1] called numFrames or nFrames. when writing code be very aware of the difference between frames and samples. There's a short explanation of the difference at the top of ofSoundBuffer.h. Whenever you're talking about the size or length of a chunk of audio, be explicit somehow whether you're talking about frames or samples. Check the internals of the ofSoundBuffer class for examples.
  • [1] on ofBaseSoundStream it's called numFramesPerBuffer.

Subclassing

subclassing ofBaseSoundStream

• ofSoundStream's isSetup() calls getNumInputChannels() and getNumOutputChannels() to determine if the streaming system is setup or not. Make sure your subclass returns 0 for these immediately after construction, and that they return 0 after close().
• A lot of automatic initialisation has been added to various things, and the order of initialisation is now more flexible. This could result in repeated calls to your close() and setup() methods than in older versions of oF. Make sure you cleanly tear everything down on close(), so that you're able to set it all up again immediately afterwards on setup(). If you can't do this be very noisy about it in the logs: lots of ofLogError() methods or even an assertion or two, just to let the developer know they should be more careful. For an end-user there's nothing worse than no sound output and no log messages, simply because two function were called in the wrong order.
• After setup, after changing the ofBaseAudioOutput or ofBaseAudioInput pointers or any other times the buffer parameters might change, make a call to audioInBuffersChanged()/audioOutBuffersChanged on the input/output pointers
• Always call audioIn/audioOut(ofSoundBuffer), not the other audioIn/audioOut methods.
• Call ofBaseSoundStream::applySoundStreamOriginInfo() with the input and output buffers before calling audioIn/audioOut.
• Clear the output buffer to 0 before calling audioOut().
• Set tickCount to 0 on setup and increment once per callback call.

subclassing ofBaseSoundOutput/ implementing audioOut

• Wait for the call to buffersChanged() before setting up your internal buffers. Ensure you can cleanly tear down and setup again if buffersChanged() is called twice with different settings.
• Until we have an internal FIFO on the ofSoundStream, audioOut is called on the sound card's timer thread, not the main thread. Make sure your audioOut is threadsafe but try not to do things that might block for a long time, like file I/O. If you need to do a lot of work, do it in a different thread that writes to a lock-free FIFO and read from this FIFO in audioOut. See Ross Bencina's guide for tips.
• audioOut()'s float* output comes pre-cleared: audioOut does not need to set to 0 if it doesn't render anything

Design

• Sound programming design

Todo

✓ = done

Testing

damian is developing on OSX

• test iOS
• test Linux
• test Windows
• test Android

ofSoundMixer

• ✓ damian move ofMixer instance out of ofBasicSoundPlayer, make singleton
• ✓ damian deal with singleton/root ofSoundStream instance: need some way of selecting the output at startup
  • (arturo) i would try to avoid singletons they give lots of headaches, instead every ofSoundStream should create it's own mixer then you can chain other mixers if needed. that way you can add more than one out to a soundstream transparently and it'll add them to it's internal mixer
  • (damian) IMO that couples the sound stream too much to the mixer, and you'll end up with duplicate code, for example setting the routing options for the mixer, especially once the mixer allows more advanced routing options like aux busses. i think it's safe to assume that there's only one ofSoundStream and that ofSoundStream has an ofSoundMixer associated with it, but for the majority of users it will be enough (and less scary) to say ofSoundMixerGetSystemMixer()->setPan(myCustomofBaseSoundOutputSubclassWithoutItsOwnPanning, 1); anyway have a look at my branch as it is now, the ofSoundPlayer example -- it has ofSoundMixerGetSystemMixer() and ofSoundStreamGetSystemStream(), but also allows anyone to define their own mixer or sound stream and use that instead. tell me if you think it's headachey or not.
  • (arturo) won't it be still coupled if by default it autoconnects and creates a soundstream with default parameters? i think the normal logic is to create a sound stream with the parameters you want and add a mixer to it then generators to the mixer, to make things easier the soundstream can have a mixer in it, since the mixer doesn't have parameters it's ok to create one by default in every soundstream but making the mixer create a soundstream with default parameters seems kind of weird. perhaps let's not do defaults by now only the classes and see what it's the most useful. also if you create singletons like that it's better to use ofPtr instead of normal pointers.
  • (damian) agreed it's a bit weird for the ofMixer to setup the soundStream. ok, i'll shift the soundStream automatic setup to the first call to setOutput() or setInput() (the system ofSoundMixer sets itself up on the first call to addOutput). i realise the normal logic is stream then mixer then output, but it doesn't need to be: i should be able to just setup the output and if i don't care about the details the mixer and stream should set themselves up if they haven't been setup already, this makes it easier to use.
  • (damian) and further; if i just want a mixer and don't care about the sound stream, in this case i can call ofSoundMixerGetSystemMixer() and immediately call addOutput to that mixer, without having to think about whether i've setup the ofSoundStream first. really, this is one of the things that has irritated me and been flaky about oF's sound system for so long, is that the order of execution at the setup phase makes a difference to whether or not things works later. seriously, we can make the code smarter than that!
  • (damian) also we seem to have different understandings about what 'decoupled' means in this context... my ideas for the mixer require that it can exist separately from the ofSoundStream, it's not just a simple class that bounces down a bunch of inputs to one output, it also can do other things like routing channels. my instincts say that to bury this functionality inside ofSoundStream will cause pain later on.
• allow ofSoundMixer to change its output node: change setup() to take ofBaseSoundOutput as output target.

ofBaseSoundStream, ofBaseSoundOutput subclasses

• ✓ damian implement buffersChanged() method (called when buffer sizes changed, before audioOut is called: subclasses should overload this if they have an internal buffer that needs to be allocated to match the buffer size in audioOut())
• ✓ damian migrate all subclasses to use audioOut( ofSoundBuffer& buffer ) instead of audioOut( float*, int, int )
  • this might need tickCount as an additional parameter, it could even be in the sound buffer as something that only soundstream can set (friend?) and others can read
• change all testApp (examples, templates, ...) to use audioOut( ofSoundBuffer& buffer )
• ✓ damian refactor bufferSize->nFrames

ofSoundStream

• implement FIFO in ofSoundStream, default to calling audioOut on the main thread (increasing latency but making audio more robust and avoiding thread-safety issues) ref (https://github.com/mohamed/lock-free-fifo)
  • you mean for calling audioOut/audioIn? seems weird sound always run in it's own thread, if the opengl thread takes slightly long to run you'll get buffer underuns really easy
    • (damian) right, the problem is there are thread safety issues, for example you can't just add/remove channels to a mixer without locking, and locking can potentially also cause underruns. also naive downstream users of the ofSoundStream api could very easily create weird audio artefacts or crashes if they for example resize or resample a buffer while it's being played.
• allow users to switch audioOut calls from main thread to their own thread (for low-latency applications)
• allow separate input and output device IDs
  • this is already there creating 2 soundstreams but i think the default should be the same device, if you want to do something where you get the input process it and output it having separate devices can be a headache that's why rtaudio or portaudio open in and out in the same call
    • (damian) hmm, no it's not, there were two sound streams but only one was actually being used. actually with RtAudio, the OSX system sound is exposed as two separate devices. If you install Soundflower then call ofSoundStreamGetSystemStream()->setDeviceID(2), audio is routed to Soundflower. But if you then call ofSoundStreamGetSystemStream()->setDeviceID(0) it fails with an error if nOutputChannels>0, while setDeviceID(1) fails with an error if nInputChannels>0.
• ✓ damian preserve inputPtr and outputPtr between calls to setup() for cases where ofSoundStreamGetSystemStream()->setup() is called after the mixer has already been created and plugged in to the ofSoundStreamGetSystemStream().
• ✓ damian shift ofSoundStreamGetSystemStream() setup out of the ofSoundMixer and into the first call to setOutput()/setInput()

ofSoundBuffer

• ✓ damian refactor bufferSize->nFrames
• inspection methods? ofSoundBuffer::drawSpectrum()?
  • seems too much to me
• FFT methods
  • there's fft implementation using kiss in ofOpenALSoundPlayer we should refactor it
• ability to lock a buffer's size and format parameters? eg for audioOut() where we don't want subclasses resizing the buffer or changing channel count

ofSoundFile

• test libaudiodecoder streaming on iOS/OSX
• build libaudiodecoder on Windows, test

ofVideoPlayer

• optionally route audio from ofVideoPlayer through the system ofMixer

ofBasicSoundPlayer

• allow audio output to be sent to a custom (non-system) ofSoundMixer instance
  • issue: for ofBaseSoundOutput instances to know which mixer they're on, we'd need a doubly-linked structure (mixer->baseSoundOutput, baseSoundOutput->mixer and related connection/disconnection methods), this would allow myPlayer.setMixer(newMixer) and myPlayer would automatically disconnect from oldMixer. otherwise we're forced to do oldMixer->removeOutput(myPlayer); newMixer->addOutput(myPlayer);. But, ofSoundPlayer is not a subclass of ofBaseSoundOutput, only ofBasicSoundPlayer is. But, (eg) ofFmodSoundPlayer cannot inherit from ofBaseSoundOutput. Halp?
  • ref: https://github.com/damiannz/openFrameworks/commit/28b436962435337e12638c8c56fac81c2cee5fbb#commitcomment-1941021
  • --> create a doubly linked structure (ofBaseSoundOutput objects have a link back to their downstream objects, eg mixers)

ofSoundMultiplexor

• implement. allows a single ofBaseSoundOutput to be an audio source for multiple downstream sinks. do this by: rendering once into a buffer, copying from that buffer to each downstream as requested. needs tickCount parameter.

platform-specific

• iphone sound player: read-back actual value of nFramesPerBuffer after setting it. ref (http://stackoverflow.com/questions/4863330/adjust-the-length-of-an-audiounit-buffer)
• android sound player: refactor bufferSize->nFramesPerBuffer