Music Video Similarity

P2 Multimodal Information Processing and Analysis project - MSc in Data Science @ NCSR Demokritos
Find the most similar song in database

Steps:

1. Compile a list of 1000 videos along with metadata attributes

~ Create a 10 genre list
~ Implement a script for:

• Downloading top 50 youtube videos per music genre
• Gathering metadata for each video on spotify
• Saving video metadata and file attributes in a Postgres database

2. Extract audio and video features

• Split videos into 10sec segments
• Extract audio features using pyAudioAnalysis
• Extract video features using Tyiannak modules
• Save audio & video features in database
• Train a NN LSTM autoencoder (44 timesteps x 422 features) for encoding videos into a fixed length vector (40 dimensions)
• Train a NN classifier on database video 40D encodings for predicting new video genre (10 target classes)

3. Compute similarity graph for audio, visual and audio-visual feature representations

• Implement KNN on SQL evaluating distance weights along each feature
• Use Euclidian and Cosine similarity metrics

4. Evaluate similarity in terms of metadata

• Classify the new video by the pre trained classifier, boost same genre videos on similarity

Training History & Confusion Matrix

Running the app

The app is built with Flask framework. A Postgres container serves the database, therefore Docker and the psycopg2 driver are required to be installed.

• Download and extract database data into the project root directory.
• Install the project requirements:

$ pip3 install -r requirements.txt

• Run the app by sourcing run.sh

$ source ./run.sh

• Open your browser and connect to http://localhost:5000

Search flow