Viewvie is able to ascertain the name of the uploaded video, by matching its frame embeddings to what is stored in its database. The metadata of the most prominent matches are returned to the user, telling them the name of the video.
Unfortunately, for now we face optimization issues, and have limited our testing (and use) to short clips of around 2-5 seconds. We recommend you stay within this range as well when trying it locally on your system.
This project is also very early in development. =)
Here we have two videos, the first which was upserted to our Vector Database with metadata containing its title, and the second which was not upserted.
On the front-end, querying both videos yields the following results respectively:
For the first video, the application correctly predicted the name of the video, as it was present in the dataset. For our second video, it correctly ascertained that there was no match for the video, as it was not in the database. Feel free to try your own inputs for the querying, but only the ones in the test-input directory, or any frames (you can upload images) or snippets from the same videos will yield any results.
Here is also an interesting graphical representation of the current vectors in the Pinecone Database! It was made using Atlas Nomic.
Just a disclaimer, unfortunately, in the face of our time (and budget) constraints, we do not have support for Mac or Linux users. This is because I say so. (Unfortunately like a shortsightedly architectured house, I only have Windows.)
Let's get started.
First, fork this repo, so you can have your own personal playground.
Then, use git clone <forked_repo_link> in the terminal of your preferred IDE.
In order to get up and running, you must have NodeJs installed. You can download the latest version here. This installation will include the Node Package Manager (npm).
You can check if NodeJs and npm are installed correctly by running node -v and npm -v respectively. If it does not recognize either as a command, check if Node is set as a PATH environmental variable. If it is not, set Node as a PATH environmental variable.
Once Node is set up, let's get our project dependencies set up with npm run install-all.
That covers our NodeJs dependencies. Time for our Python dependencies.
Now, open your terminal. You can do this by hitting Win + R, typing 'cmd' and hitting enter.
Run python -m venv viewvie in your CLI to initialize the virtual environment.
This will create a virtual environment file (named viewvie) in your user directory.
Use viewvie\Scripts\activate in your CLI to activate the virtual environment. If your commands (even before your C: directory) start with (viewvie), the environment has been set up correctly!
This is where we will install dependencies for our embedder module.
Install the python dependencies by running each one in your CLI:
pip install mediapipe
pip install opencv-python
After these packages have installed, you can type deactivate in the CLI to deactivate the virtual environment.
Now, we can safely run npm run start-all from the 'viewvie' directory to start the application! You will see a http://localhost:5173/ in your console - click it to open the application in your browser.
Of course, project is missing .env file with Pinecone API key. You can set this up by adding a .env file to the 'backend' directory, and adding PINECONE_API_KEY='<actual-api-key-not-this-this-is-not-an-api-key>' to it. Contact me for details on the Pinecone API key. =)
For now, we have a very limited pool of movies in our database. There is functionality in the application to add more, but we want to straighten out load balancing first before we let users test this out. Temporarily, we will provide sample video clips to test the applications. They will be labeled with the actual movie name (which you wouldn't actually know). Some will match movies in our database, and some will not. They can be downloaded in the 'test-input' directory.