Document Model

Document model is where Source will store data.

This will chunk the data later on into DocumentChunks but for now, pre-tranformations it stores what we need to continue to the transformation step

Status Field

Just to see if it is still running, or complete (Might go away)

Was here to show the cost of embedding

This can be a URL, filename etc. This will help the Transformers.

Initial dump of content if any. For example a PDF file might be null till we parse it with a transformer

This is the Source model and this is what links it to the Type of data we are dealing with as well. This also links it back to the project.

Might go away soon