feat: vector store #68

TimPietrusky · 2024-03-07T22:45:29Z

Motivation

Integrates a vector store (Qdrant)
Runs Qdrant locally
Starts the vector store when the app is started
Added qdrant-win.7z into the installer

Discuss

Needs an OpenAI API Key for the embedding, which doesn't exist yet when this is started for the first time, so we need to come up with an idea on how to solve this once the API Key is there. Currently we just set the API Key "none"

.eslintrc.json

pixelass · 2024-03-08T12:00:20Z

jest.config.electron.ts

@@ -4,6 +4,7 @@ const jestConfig = {
 	...defaults,
 	roots: ["<rootDir>/src/electron"],
 	testMatch: ["**/?(*.)test.ts"],
+	testPathIgnorePatterns: [".e2e."],


this suggests the naming pattern of e2e to be *.e2e.test.ts

This is a regex pattern, which makes sure to work for both directories like __test__/e2e/* and files *.e2e.test.ts.

And the purpose is to ignore any files / folders that contain e2e.

pixelass · 2024-03-08T12:01:07Z

jest.config.electron.e2e.ts

+const jestConfig = {
+	...defaults,
+	roots: ["<rootDir>/src/electron"],
+	testMatch: ["**/e2e/**/*.test.ts"],


shoul this rather be *.e2e.test.ts? (see comment beolw)

I decided against having e2e in the file name, but have a dedicated folder for e2e-tests.

src/client/pages/[locale]/installer/02.tsx

src/electron/future/main.ts

src/electron/future/services/vector-store.ts

pixelass · 2024-03-08T12:13:36Z

src/electron/future/utils/path-helpers.ts

@@ -12,6 +12,8 @@ export const resourcesDirectory = isDevelopment
 	? path.join(process.cwd(), "resources")
 	: path.join(app.getPath("exe"), "..", "resources", "app.asar.unpacked", "resources");

+export const userDataDirectory = isDevelopment ? process.cwd() : app.getPath("userData");


can we add a dedicated folder here which we can then ignore. This approach won't scale as it could poolute the root directory

actually, userData is already handled. it uses either captain or captain__development__

My problem here is, that I can't use app.getPath, as this is something that only exists in electron. For testing I want to run my e2e tests without electron, as this allows me to directly execute everything and don't interact with my code via IPC. So both of these folders are not reachable for my tests at all.

This approach is also not poluting the root, as the folder is .gitignored, so it will only be relevant for a person that wants to test this.

This solution was something I came up with after trying many other things and all of them failed. But I will try one last thing and mock the app.getPath folder for this specific test.

It worked, I mocked app.getPath with process.cwd() for my specific test. I removed all the other changes to path-helpers.

pixelass

Please see comments

pixelass · 2024-03-08T12:17:10Z

src/electron/future/utils/path-helpers.ts

@@ -25,7 +27,7 @@ export function getDirectory(...subpath: string[]): string {
 }

 export function getUserData(...subpath: string[]): string {
-	return path.join(app.getPath("userData"), ...subpath);
+	return path.join(userDataDirectory, ...subpath);


see above. We cannot do this.

Was removed, see explanation above.

Co-authored-by: Gregor Adams <1148334+pixelass@users.noreply.github.com>

src/electron/future/services/vector-store.ts

# [1.0.0-alpha.10](v1.0.0-alpha.9...v1.0.0-alpha.10) (2024-03-08) ### Features * vector store ([#68](#68)) ([2dbe978](2dbe978))

TimPietrusky · 2024-03-08T14:48:24Z

🎉 This PR is included in version 1.0.0-alpha.10 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

TimPietrusky and others added 9 commits March 3, 2024 20:31

feat: vector store with langchain & hnswlib

a4a4e46

feat: use qdrant as vector store

9d1b109

feat: integrated verctor store everywhere

7c3c046

fix: renamed from assistant to vector-store

ceb39a9

Merge branch 'alpha' into feat/vector-store

61d8717

refactor: renamed type to "VectorStoreDocument"

76b17a0

refactor: remove unused constant "ASSISTANT"

1705980

feat: added "qdrant-win.7z" to the installer

95aaf1c

test: updated tests to work with "userDataDirectory"

0c9a1d5

TimPietrusky requested a review from pixelass March 8, 2024 11:14