Surround Sound

[drowaudio]

Audio input/output busses

• Currently input/outputs are created from either single channels or “treat as stereo pair” stereo channels. Either replacing this paradigm or in addition to, there should be the ability to create an input/output bus and then assign IO channels to this using a matrix
• This should be at the global/Engine level so apply to all WaveInput/OutputDevices
• It should be possible to assign IO channels to multiple busses
• There should be some quick config to assign all to stereo pair busses, 5.1, 5.1.4 etc. starting from a specific channel (this might simply be a UI task though)
• This effectively means modifying WaveInput/OutputDevice so getChannels/getChannelSet/consumeNextAudioBlock work correctly
• Once this is done, it should be possible to assign a TrackOutput to one of these WaveOutputDevices and assign a

Audio graph

• The audio graph can already deal with multiple channels, it just doesn’t have any notion of what those channels “mean”. In theory if a leaf Node (like WaveNode) reports it has 6 channels, all the above Nodes will work with 6 channels so the WaveNode gets 6 channels to fill
• Ignoring the intermediate Nodes, you have leaf WaveNodes that end up putting putting their channels in to a channel of a WaveOuputDevice, ignoring the meaning of the channel. This sounds like the most sensible approach to me and it should be up to special “mixing” Nodes that can convert between input channel layouts to a destination channel layout
• WaveNode does a bit of this internally as you can supply a multi-channel audio file but limit its reading to 2 channels
• I think NodeProperties needs to have an juce::AudioChannelSet (or similar) member which Nodes can use to describe their buffers. As this passes through the audio graph, this information will either simply be passed (if they don’t care about the layout) on or changed (if they’re remapping) by Nodes.
• Again, in the simple case, this means trackiton_EditNodeBuilder.cpp createNodeForDevice will be able to create a ChannelMixUpDownNode (which doesn’t currently exist) to map from the input channel format to the WaveOutputDevice’s format if required.
• Additionally, Plugin subclasses could do this remapping with a given mix up/down formula
• The aim for the Engine should be to do the least surprising thing and leave customisations up to client code. This might well mean just dropping additional channels or zeroing them out depending on the number of up/down channels
• WaveInputDeviceNode might need tweaking to ensure it fills the correct channels with its info. It might need to report the channel layout in getNodeProperties

Audio clips/file reading

• At the moment AudioClipBase has a getActiveChannels member which returns the configuration of the clip. This may need to be updated when a clip is added as at the moment it’s only really used to set left/right/stereo configurations
• This gets passed to the WaveNode to tell it what channels to read out of the audio file
• AudioFileCache::Reader::readSamples has some logic to try and map channels when reading but this might be incorrect. You would expect the Clip’s channel layout to be the same as the file’s but if they’re set to something different, I’m not sure it’s the file reader’s job to try and map between them? I would have thought that AudioClipBase::activeChannels should really be a bitmap stating which of the source file channels are being used. This then gets flattened in to a set of channels with the corresponding layout. E.g. if you have a 7.1 file the clips activeChannels should be [L R C LFE Ls Rs Rls Rrs] but you might only enable the following [L R C LFE Ls Rs X X] to create a 5.1 clip. This is effectively how L/R channels are created from a stereo file but I’m not sure if that scales to more channels and if there is a standard way to mix between formats?

Mix up/downs

• As I’ve mentioned a couple of times, I’m not too familiar with how and at what points mix up/downs should happen. The most thorough thing I’ve seen is the DADman/DADmon software that provides mappings between each format and the gains to be applied etc. We might need to create these as well so they can be used at various points. We might also have to have a way for these to be user-specified as I’m sure we won’t be able to deal with every possible combination
• Once we have the above, it should be possible to create a Plugin which you can use to convert between channel layouts
• I’m not quite sure what to do where there are ambiguous mix points. E.g. you have a stereo output bus but a mono file that has a centre channel layout. Do you ignore the layout and map the source C[0] to destination L[0] or duplicate to both L[0] & R[1]? Or maybe just ignore the channel completely?
• A final feature could be to add a surround mixer Plugin. I’m not sure how these work but presumably it would involve selecting a channel, treating it as a mono source and then moving it in a 2D space to add it to a set of destination channels. I’m not sure if you can pan multi-channel signals like this or if there are standard algorithms.

[jdieks]

Hey David. Thanks for looking into this. I cannot comment too much on the real in depth technicalities, I hope the rest of our team will chime in.
About the conversion points:
https://youtu.be/kUTp8ubV4kE?t=149
So Steinberg for instance have a channel/format converter plugin that they use for this, with many presets like the Lo/Ro one I was talking about. They automatically place this plugin on points where downconversion is needed.

Also, when you don't use any 2.0+ busses in the project, all panners are basically either 1 stereo panner, 1 "waves like" panning/width or 2 mono panners

Once you start adding 2.0+ busses in the project, the gfx of all panners change, where the backend of stereo/mono channels that are send to stereo/mono busses are still the same, but once a track is send to a 2.0+ bus, the gfx is replaced by a surround panner.

When you click this surround panner you'll notice that in the backend it's "just" a VST that has been auto inserted and that comes up, allowing for a lot more (fine) control.

Indeed, the way I understand it, these panners work by taking all the input tracks as mono tracks on the input and then "leveling" and "mixing" them into the output tracks.

I have not spend a lot of time looking into the following yet, but there is a github repo with these panners and upmixers made in reaper script:
https://github.com/junh1024/Reaper-Surround

[drowaudio]

Ok, I think we'll leave the panning and up/down mixing for actual Plugins and just treat all the channels as a list everywhere else or it's going to be too confusing to actually build a graph.

Audio clips/file reading

• When a new WaveAudioClip is inserted, ensure activeChannels gets defaulted to the layout of the source file rather than just a stereo output
• Add a check to WaveAudioNode (and WaveAudioNodeRealTime) to ensure the destination channel layout is a subset of the source file channel layout. This should avoid having to do any mix up/downs at the file reading stage.
• Ensure the logic in AudioFileCache::Reader::readSamples which maps channels to the destination output bus is used. This might mean tweaking the EngineBehaviour::isDescriptionOfWaveDevicesSupported to something more appropriate that's always enabled but is backwards compatible (with the existing left/right/stereo behaviour)

Audio graph

• Add a juce::AudioChannelSet, channelLayout member to NodeProperties. This can be implemented for Nodes to return their channel layout
• Add an assertion in Node::initialise to check that that channelLayout.size() is the same as props.numberOfChannels

Audio input/output busses

• Change the name of EngineBehaviour::describeWaveDevices to getWaveDeviceDescriptions and have it return std::vector<WaveDeviceDescription> rather than take the output by reference. This is where user code can return whatever bus layouts they require. If they change in their UI, they simply call engine::DeviceManager::rebuildWaveDeviceListIfNeeded() which will update the DeviceManager::getWaveOutDevice (...) etc. and the EditPlaybackContext's OutputDeviceInstances

Mix up/downs (optional)

• Add a class that can specify how to map from one format to another (essentially a matrix with settable gains at each element)
• Create a Plugin subclass that can have a conversion matrix set and applies it to destBufferChannels member of the PluginRenderContext (although this should probably be added to the PluginInitialisationInfo struct instead to indicate it won't change)
• Add a surround mixer plugin (to be specified)

---

So it doesn't look like there's actually too much to do as the basics are already there, it's really just a case of exposing the channel info in several places. I've left the multi-channel input busses side of things for now as I need to have a think about backwards compatibility and recording file formats but again this shouldn't be too hard. It just needs some modification to WaveInputDeviceNode::acceptInputBuffer and AudioFileWriter to use a juce::AudioChannelSet