SyncSink – Synchronize media files

SyncSink is a tool to synchronize media files with shared audio. SyncSink matches and aligns shared audio and determines offsets in seconds. With these precise offsets it becomes trival to sync files. SyncSink is, for example, used to synchronize video files: when you have many video captures of the same event, the audio attached to these video captures is used to align and sync multiple (independently operated) cameras. Evidently, SyncSink can also synchronize audio captured from many (independent) microphones if some environmental sound is shared (leaked in) the recordings.

Fig. Synchronizing audio with the SyncSink user interface.

SyncSink has also uses to synchronization of data-streams. For those applications please see the article titled Synchronizing Multimodal Recordings Using Audio-To-Audio Alignment.

How does SyncSink synchronize media files?

The first step is to extract fingerprints from a reference media file. Subsequently fingerprints are extracted from other media files and a rough offset is determined. The rough offset determines how much the ‘other’ file needs to shift to match the reference. The rough offset is accurate to about 8ms.

To improve the rough offset crosscovariance between the refrence and other audio is calculated. Since we already have a rough offset we know where audio is likely to match so we can reduce the amount of crosscovariance calculations, which are computationally intensive. In the ideal case the crosscovariance is stable and improves the offset up to audio-sample accuracy.

How to use SyncSink

First you will need to install ffmpeg and a recent JRE. SyncSink decodes audio using by calling ffmpeg. On a Debian like system you can use apt-get to install ffmpeg, brew on macOS is similar:

apt-get install ffmpeg

After installing ffmpeg and a JRE, SyncSink can be started by double clicking the latest JAR file. If that does not work you can use the command line: java -jar syncsink.jar.

Once started, add various audio or video files using drag and drop. Similarly to the screencapture above. If the same audio is found in the various media files a timebox plot appears with a calculated offset. To sync the media files, pres the Sync! button. In the messages window ffmpeg commands can be found to syncronize the media files. The commands are also printed on standard out.

Further Reading

Some relevant reading material concerning SyncSink.

1. Six, Joren and Leman, Marc “Synchronizing Multimodal Recordings Using Audio-To-Audio Alignment” (2015)
2. Six, Joren and Leman, Marc Panako – A Scalable Acoustic Fingerprinting System Handling Time-Scale and Pitch Modification (2014)
3. Wang, Avery L. An Industrial-Strength Audio Search Algorithm (2003)
4. Ellis, Dan and Whitman, Brian and Porter, Alastair Echoprint – An Open Music Identification Service (2011)
5. Sonnleitner, Reinhard and Widmer, Gerhard Quad-based Audio Fingerprinting Robust To Time And Frequency Scaling (2014)

Credits

The SyncSink software was developed at IPEM, Ghent University by Joren Six.

If you use the synchronization algorithms for research purposes, please cite the following work:

@article{six2015multimodal,
  author      = {Joren Six and Marc Leman},
  title       = {{Synchronizing Multimodal Recordings Using Audio-To-Audio Alignment}},
  issn        = {1783-7677},
  volume      = {9},
  number      = {3},
  pages       = {223-229},
  doi         = {10.1007/s12193-015-0196-1},
  journal     = {{Journal of Multimodal User Interfaces}}, 
  publisher   = {Springer Berlin Heidelberg},
  year        = 2015
}