upload sources file error "I/O operation on closed file"

[hqzhon]

Describe the bug
upload sources file error "I/O operation on closed file"

Please describe your setup

• How did you install memgpt?
  • pip install pymemgpt-nightly and git clone
• Describe your setup
  • ubuntu 22.04
  • Terminal

run logs
INFO: 127.0.0.1:42014 - "GET /api/agents HTTP/1.1" 200 OK
INFO: 127.0.0.1:42014 - "GET /api/sources HTTP/1.1" 200 OK
INFO: 127.0.0.1:42014 - "POST /api/sources/af96f667-f50d-480b-96a9-34324bc0c4ab/upload HTTP/1.1" 200 OK
I/O operation on closed file.

MemGPT Config
~/.memgpt/config file :
[defaults]
preset = memgpt_chat
persona = sam_pov
human = basic

[model]
model = wizardlm2:7b
model_endpoint = http://10.60.84.95:11434
model_endpoint_type = ollama
model_wrapper = chatml
context_window = 8192

[embedding]
embedding_endpoint_type = hugging-face
embedding_endpoint = http://10.60.84.95:11434
embedding_model = nomic-embed-text:latest
embedding_dim = 384
embedding_chunk_size = 300

[archival_storage]
type = chroma
path = /home/cm/.memgpt/chroma
uri = postgresql+pg8000://memgpt:memgpt@localhost:5432/memgpt

[recall_storage]
type = sqlite
path = /home/cm/.memgpt
uri = postgresql+pg8000://memgpt:memgpt@localhost:5432/memgpt

[metadata_storage]
type = sqlite
path = /home/cm/.memgpt
uri = postgresql+pg8000://memgpt:memgpt@localhost:5432/memgpt

[version]
memgpt_version = 0.3.14

[client]
anon_clientid = 00000000-0000-0000-0000-000000000000

[pavel604]

I have the same problem.

For some reason, in rest_api/sources/index.py, method load_file_to_source receives the file already closed. I was able to get around it by reading bytes in enclosing method upload_file_to_source, and passing them through add_task as below:

def load_file_to_source(server: SyncServer, user_id: uuid.UUID,  source: Source, job_id: uuid.UUID, 
    file: UploadFile, 
    bytes: bytes <------
    ):
    # update job status
    job = server.ms.get_job(job_id=job_id)
    job.status = JobStatus.running
    server.ms.update_job(job)

    try:
        # write the file to a temporary directory (deleted after the context manager exits)
        with tempfile.TemporaryDirectory() as tmpdirname:
            file_path = os.path.join(tmpdirname, file.filename)
            with open(file_path, "wb") as buffer:
                buffer.write(bytes) <-----

and then

    async def upload_file_to_source(
        # file: UploadFile = UploadFile(..., description="The file to upload."),
        file: UploadFile,
        source_id: uuid.UUID,
        background_tasks: BackgroundTasks,
        user_id: uuid.UUID = Depends(get_current_user_with_server),
    ):
        """
        Upload a file to a data source.
        """
        interface.clear()
        source = server.ms.get_source(source_id=source_id, user_id=user_id)
        
        bytes = file.file.read() <------
        # create job
        job = JobModel(user_id=user_id, metadata={"type": "embedding", "filename": file.filename, "source_id": source_id})
        job_id = job.id
        server.ms.create_job(job)

        # create background task
        background_tasks.add_task(load_file_to_source, server, user_id, source, job_id, file, bytes) <----