🚀 Supercharge your LM Studio with a Vector Database!

Now with the ability to process images and audio files, Local chat models, and text to speech playback!
Introductory Video
Medium Article

	Database	Transcribe	Bark/TTS	Vision
Intel CPU	✅	✅	✅	✅
AMD CPU	✅	✅	✅	✅
Nvidia GPU	✅	✅	✅	✅
AMD GPU on Windows	❌	❌	❌	❌
AMD GPU on Linux	✅	❌	✅	✅
Apple CPU	✅	✅	✅	❌
Apple Metal/MPS	✅	❌	❓	❌

• 🔥Due to time constraints I'm currently unable to support all backends/platforms; therefore, this table is only current as of Release v3.5.2. Any release after v3.5.2 is only guaranteed to support the Windows + Nvidia GPU combination. My goal is to also support the Windows + No Nvidia GPU setup. However, I will not have time to test this and will SOLELY be relying on users submitting an issue in GitHub to identify bugs. However, I will address any bugs submitted as a GitHub issue.

REQUIREMENTS

1. 🐍Python 3.10 or Python 3.11

2. 📁Git

3. 📁Git Large File Storage.

4. 🌐Pandoc.

5. Build Tools.

   Certain dependencies don't have pre-compiled "wheels" so you must build them. Therefore, you must install something that can build source code such as Microsoft Build Tools and/or Visual Studio. If you decide to use both of these programs in conjunction, make sure to select the "Desktop development with C++" extension and check the four boxes on the right containing "SDK."

   EXAMPLE ERROR ON WINDOWS

   
   EXAMPLE SOLUTION ON WINDOWS

   

6. 🟢Nvidia GPU acceleration requires CUDA 12.1+.

INSTALLATION

🪟WINDOWS INSTRUCTIONS

Step 1

🟢 Nvidia GPU ➜ Install CUDA 12.1+

🔴 AMD GPU - (AMD GPU acceleration support coming in future releases).

Step 2

Download the ZIP file for the latest "release," extract its contents, navigate to the "src" folder to run the commands below.

• NOTE: If you clone this repository you WILL NOT get the latest release. Instead, you will development versions of this program which may or may not be stable.

Step 3

Navigate to the src folder, open a command prompt, and create a virtual environment:

python -m venv .

Step 4

Activate the virtual environment:

.\Scripts\activate

Step 5

Run setup:

python setup.py

Optional Step 6

Run this command if you want to doublecheck that you installed the Pytorch and gpu-acceleration software correctly:

python check_gpu.py

🐧LINUX INSTRUCTIONS

Linux users must use Release v3.5.2 until I can update the codebase due to recent major changes. Download the ZIP file for that release and follow the instructions in the readme.md.

🍎APPLE INSTRUCTIONS

MacOS users must use Release v3.5.2 until I can update the codebase due to recent major changes. Download the ZIP file for that release and follow the instructions in the readme.md.

USING THE PROGRAM

🖥️INSTRUCTIONS🖥

Activate Virtual Environment

• Once you install the program you've already created a virtual environment, so you just need to activate it each time you want to restart it. Remember to run the appropriate command to do so (based on your platform) within the src folder.

Start the Program

python gui.py

Only systems with an Nvidia GPU will display gpu power, usage, and VRAM metrics.

🔥Important🔥

• Read the User Guide before sending me questions.

Download Vector Model

• In the Models Tab tab, choose the embedding model you want to download. The User Guide Tab contains information about the various models.

Create a Vector Database

• In the Create Database tab, click Choose Files and select one or more files to add. This can be repeated as many times as you wish.
  • Supported documents are: .pdf, .docx, .epub, .txt, .html, .enex, .eml, .msg, .csv, .xls, .xlsx, .rtf, .odt.
• If you selected any image files, I highly recommend that you adjust the vision model settings within the Settings tab and testing the settings while processing a single image, which can easily be done within the Tools tab.
  • Supported images are: .png, .jpg, .jpeg, .bmp, .gif, .tif, .tiff
• 🔥 To add audio files you must transcribe one or more audio files from the Tools Tab. The transcriptions will be saved and added when you create the vector database.
  • Supported audio extensions include, but are not limited to: .mp3, .wav, .m4a, .ogg, .wma
• Click Choose Model.
• Click 🔥ONCE🔥 on the directory containing the vector model you want to use and then click "OK."
• In the input box, enter a name for the database you want to create (accepts numbers, lowercase letters, and the characters "-" or "_").
• In the Settings Tab, set the chunk size, chunk overlap, and the device you want to use. More information is in the User Guide.
• Click the Create Vector Database button.
  • 🔥 MAKE SURE to wait until the command prompt states that the database has been successfully created before proceeding.

Delete a Database

• In the Manage Databases tab, select a database from the pulldown menu and click Delete Database.

  The ability to delete one or more specific files is coming soon.

Query a Database (No LM Studio)

• In the Query Database tab, select the database you want to use from the pulldown menu.
• Enter your question by typing it or using the Record Question button.
• Check the chunks only checkbox.
• Click Submit Question, which will return chunks relevant to your query.
  • In the Settings tab, you can change multiple settings regarding querying the database. More information can be found in the User Guide.

Query a Database with a Response From LM Studio

This program can get relevant chunks from the vector database and forwarding them - along with your question - to LM Studio for an answer!

• Perform the above steps regarding entering a question and choosing settings, but make sure that Chunks Only is 🔥UNCHECKED🔥.
• Start LM Studio and go to the Server tab on the left.
• Load a model. I've personally tested the following models as good:
  • Marx_3B_V3
  • Mistral_7B_Instruct_v0_2
  • Neural_Chat_7b_v3_3
  • Llama_2_13b_chat_hf
  • SOLAR_10_7B_Instruct_v1_0
• Turn Apply Prompt Formatting to "OFF."
• On the right side within Prompt Format, make sure that all of the following settings are blank:
  • System Message Prefix
  • System Message Suffix
  • User Message Prefix
  • User Message Suffix
• At the top, load a model within LM Studio.
• On the right, adjust the GPU Offload setting to your liking.
• Within my program, go to the Settings tab, select the appropriate prompt format for the model loaded in LM Studio, click Update Settings.
• In LM Studio, click Start Server.
• In the Query Database tab, click Submit Question.

Request a Feature or Report a Bug

Feel free to report bugs or request enhancements by creating an issue on github or contacting me on the LM Studio Discord server (see below link)!

CONTACT

All suggestions (positive and negative) are welcome. "bbc@chintellalaw.com" or feel free to message me on the LM Studio Discord Server.