Audio tracks in ExternalReference?

[paulharter]

When adding a clip to an audio track how do specify which audio track/channel in the source file to use? I couldn’t see anything in the docs, examples or looking through the code. Must all media files contain a single stream like OP-Atom? Are there conventions for this in metadata on the ref object?

Sorry if I’m missing something obvious.

[meshula]

Hi Paul, that's a really great question. There's no convention or schema in place yet. On productions that have multitrack wav's, it's typical to create wav files with say 48 tracks, and then make selects across the 48. That suggests that the right way to encode this might be, in the case of wav's, a list, e.g. [3,13,14,28]. I have to imagine the conventions might be different for other media types; a mov might have more than one audio track, and those audio tracks might themselves have channels; mov's with multicam tracks are another variant. Would that imply a list of lists, one for the track selected, and another for the channels? How about AAFs?

This isn't much of an answer, just a few things that occur to me that need to be considered.

[meshula]

I'm in principal in favor of an explicit schema annotation. I'd want media-track (for example) instead of track, since track already has an explicit meaning in the schema. Numbering channels seems safe. A list of channels seems necessary to describe a single stream, one might want to pick L, R, and voice out of a 7.1 file. I'm not aware of any channel naming schemes, as far as I can tell every DAW requires you to manually assign tracks to channels in a non-portable way.

In the case of a multitrack mov file, I think, but could be mistaken, that you can request the "third audio track", but there's no "third track" per se because there's no overall ordering of tracks across track kinds. If I've got that right, then the annotation would also have to specify track kind so we know which group of tracks to count within.

Would such an annotation also imply a sanity-checking tool that would allow offline scrubbing when you get a delivery ("otio-media-check reports: file XXX.otio references YYY.mov video track 2 but there is only 1 video track"), and before going to all the trouble of importing into say Avid?

[paulharter]

I agree media-track would be better. I was thinking that this ref would always be in the context of a particular track in the timeline that would determine the type of media, but if not, yes, an explicit media-type would be necessary (and maybe good to have anyway).

I'm not sure what I feel about 0 or 1 indexing for track and channel numbers, I could go either way, 1 feels like an established notation here but my programmer brain likes 0.

Also I don't know about a sanity-checking tool, I'm not that familiar with how opentimelineio is used in people's pipelines. If there's a hook to run a py script and presumption of something like ffprobe locally then it wouldn't be too hard to do. Is there currently a check for files existence and length? Is this done with custom linkers or in the core?

Possibly the ref object could contain details about the available_streams a bit like available length, and a hash of file then the sanity check could just be to check file hash rather than introspecting file to check. This would have double benefit of checking file integrity if file has been copied or downloaded from somewhere. This would check for having the correct file but not for errors in generating the correct ref data.

I have one other thought/question about the ref

Is linking the timeline to different renderings of the same media streams possible? if so is this done by rewrites in the linker? so the uri is in effect for an abstract version of the stream rather than a particular file. In which case the hash couldn't be used as the actual file is resolved by the linker. And in this case the track and channel indexes could be different if the source media was a single mxf file with 8 audio tracks or a AAF in an edit with only one track.

[reinecke]

For visibility (#145) is the issue we have for implementing this. I think this discussion thread has already eclipsed that one in depth.

Some key points relating to this are:

• Switching between media representations (from an MOV proxy to WAV original, for instance) should be implementable by switching MediaReferences
• When channel selection is involved, swapping media references is non-trivial, it could require deeper contextual knowledge about the stream layout to ensure the expected behavior
• Does this relate at all to the multicam use case (Multicam Support #427)? Does it make sense multicam could be implemented as a media reference that points to a stack where each track becomes a stream that can be selected?
• How does this relate to Stereoscopic 3D support (Stereo (SDL) Support #150)?
• Do we want to think about how something like an HLS source might be modeled, which supplies many representations of media (video, audio, subtitle, caption), their characteristics (bitrate, language, channel layout, raster, etc.), and valid combinations of those?
• Implicit in this is panning/mixing behavior (e.x. selecting the Left channel from a stereo stream and treating it as monaural audio - like when a lav mic and boom mic are separately sent to left and right channels).

Typically, we say that MediaReference is meant to simply represent an asset somewhere and any compositional decisions should be stored on the clip level. This mental model becomes tricky when it comes to multi-stream source assets - simply swapping the media reference on a clip could imply having to repair stream selection decisions using contextual knowledge if they're stored on the clip. This pushes me a bit toward thinking of MediaReferences as a place that should store how to find a specific data stream with the desired contextual meaning on disk.

The Media fragments spec @paulharter referenced is interesting, but I think it's fraught because there are spatial and timing components that can be specified within it as well - this would imply multiple ways in OTIO to do similar things and could create confusion. It could be an interesting exercise to offer utility functions that can generate a Clip + ExternalReference from the data encoded in a media fragment URL to explore whatever solution we come up with.

I'd be interested to do some research for reference on how other systems are doing this. The FCP 7 XML spec has some channel related elements. At a glance it looks like the file element, which is roughly analogous to a MediaReference in OTIO, has a way to remap an asset's source channels into sort of a virtual output set of channels.

I think I like the idea of giving the MediaReference some first-class schema way to say "I provide a channel layout of x by selecting channels y from the source". This allows people to do things like craft media references to swap in that would present as stereo even if they're pulling form a 5.1 + LTRT source.

As the schema grows, then it could be useful to start thinking about how to specify panning/labeling output tracks. Of course, this becomes incredibly complicated if trying to model something like objects in an atmos mix.

[jminor]

I agree that channel or stream selection should be in MediaReference, not Clip. If we consider MediaReference as fully encapsulating the notion "where is the media?" or "how can I get the media?" then this makes sense. In the extreme, MediaReference could eventually be extended to pull media from all manner of complex containers, databases, etc. which should be isolated from the responsibility of Clip, which chooses which segment of the media, in time, to use.

I think multicam is a different scenario, since the collection of camera angles/takes/etc. that a multicam clip is choosing from could come from many different MediaReferences. That, to me, belongs up above the Clip level (maybe a peer to Stack, which picks just one child instead of compositing all the children).

Perhaps a good first step towards channel selection would be to extend MediaReference to have an optional "media_properties" dictionary/object which describes the resolution, sample rate, byte size, channel count, etc. of the media. In our in-house pipeline we already do this by just running ffprobe and stashing the result into a metadata dictionary. Formalizing that into the OTIO schema would make that portable and provide a basis for then specifying which channels of the media are selected or chosen to be used in the composition.

[reinecke]

I agree with @jminor about multicam modeling, so I think it's worth discarding for the purposes of this discussion. Audio channels and Stereoscopic 3D seem like the most likely use cases to consider.

I also agree with the approach of first modeling something like media_properties. It feels like the fundamental things we'd want to capture are the tracks, what channels exist in them, and some information about layout (L, R, CENTER, MONO, etc.). I haven't gotten deep into it, but I think there may be ways to attach more arbitrary metadata to channels in formats like broadcast wave, though that may be best suited for the MediaReference metadata field.

[bergner]

I spent the last few days reviewing the OTIO project as something to potentially use for integration work between editing systems, and for the most part I think it looks promising but this topic is quite concerning. Our source media would in most cases be MXF OP1A and not being able to explicitly reference a source media track feels like a severely limiting factor.

Logically I think source track selection should reside on a similar level to source_range (Clip or Item level) rather than be pushed all the way down to ExternalReference simply because any MediaReference that could "provide" multi-track content faces this problem.

It is a separate discussion how the exact media reading/demuxing is done, and that seems to be out of scope what OTIO wants to care about anyway. Like should you allow a MediaReference to be reused in the OTIO data structure because it is the same file but you are reading tracks V1, A1 and A2 and by reusing the same object you are aiming to achieve I/O optimization to only read and demux the source file once instead of 3 times. This feels like an external concern where the parent application could just compare target_url and realize that 3 MediaReference instances are pointing to the same file and some other constraints are met (e.g. timing wise in sync), and then use a single media reader/demuxer instance with some internal buffering to provide media for all those tracks.

[meshula]

Thank you for those thoughts.

As you note, mixing audio, just like image composition, is a concern of an application, OTIO should just carry enough information for the application to respond appropriately. Similarly, it's up to the application to notice if the same reference is incorporated many times within a composition.

Referring to the previous conversation, and your notes, perhaps the proposal would be to add an optional list of audio-tracks as a sibling entry to the media reference entry on a clip.

Since the majority of audio file formats don't name tracks, the optional list would have to be of either names for those file formats with named tracks, or indices.

Any notes around multicamera files should probably be referenced here ~ in case there is one-size-fits-all solution ~ or to declare one way or another that multiple audio tracks should be treated differently than multiple camera tracks within a single media file.

[bergner]

I have to admit I don't fully understand the use case for a timeline containing a track (say A1) where a Clip + MediaReference provides multiple tracks (or even more so if it is a nested Stack) and what it means to have such an entity on a single track.

Since that's on a single track in the timeline the result of processing that Clip + MediaReference (or Stack) when rendering would need to boil down to a single audio track. If not it feels like it breaks the timeline + track structure we defined in the first place because Clip+MediaReference now suddenly produce implicit extra tracks. So to me it feels like the process should involve either making a specific track selection or possibly do some kind of mixing operation that produces a single track output to ensure it "fits" within the already defined track.

[meshula]

I wasn't trying to explain such a scenario, and not trying to imply that any sort of nesting whatsoever could result.

I'll use the term "channel" as @paulhartner suggested originally, to avoid terms like "track" that already have a use.

I'm not using his suggestion of extending the clip with any other information. I'm only suggesting that "channel" could be sufficient.

Furthermore, at first blush it seems like "channel" should select a single channel. If it's necessary to reference multiple channels, further discussion is necessary, see notes below. Perhaps that concept is related to multiple references, or linking.

The channel tag would be optional, if it doesn't exist, then the existing behavior of otio would be unchanged.

AudioTrack
    --------------------------------
    Clip1
        channel 1 # zero-indexed
        MediaReference something.wav
   --------------------------------
    Clip2
       channel 0,1 # illegal
       MediaReference something.wav
   --------------------------------
    Clip3
       channel "left mic", "center mic" # illegal
       MediaReference something.wav
VideoTrack
    ---------------------------------
    Clip4
       channel "maincam"
       MediaReference something.fancy
    ---------------------------------
    Clip5
       channel "maincam", "coverage 1" # illegal
       MediaReference something.fancy

=================================
something.wav
    left channel
    right channel <--- channel
---------------------------------
something.fancy
    "left mic"
    "center mic"
    "right mic"
    "maincam"     <--- channel
    "maincam alt"
    "coverage 1"
    "coverage 2"
---------------------------------

Extra notes:

Here's the discussion of multiple media references on a clip: #630, for reference on related topics.

Here is a discussion of clips with linked audio and video. #1271 My opinion is that even though we have multiple references on a single clip, those references have a single media kind, due to the track specifying the kind. So I think that a Clip with both an audio and video reference does not fit the pattern.