Handling output files generated inside an electron

[Andrew-S-Rosen]

This is copied from #1592, which included some useful advice from @santoshkumarradha that I want to make more visible.

The original question was essentially: how do you deal with handling output files generated inside an electron such that they don't overwrite one another in the working directory?

[Andrew-S-Rosen]

EDIT: See #1628 and the PRs linked therein for an updated answer!

@santoshkumarradha's answer was:

When manually reading or writing files within Covalent, it's always a good practice to use absolute paths. By doing so, you can avoid potential confusion or issues related to file handling.

Here's an example that demonstrates the use of absolute paths in a Covalent workflow:

import covalent as ct
from pathlib import Path

@ct.electron
def job(val1, val2,file):
    with open(file, "w") as w:
        w.write(str(val1 + val2))
    return "Done!"

@ct.lattice
def workflow(input_file, val1, val2, val3, val4):
    job1 = job(val1, val2,input_file)
    return job1

input_file = Path("./example1.txt").absolute()

dispatch_id = ct.dispatch(workflow)(input_file, 1, 2, 3, 4)
result = ct.get_result(dispatch_id, wait=True)
print(result)

For chemistry codes that generate output files during calculations, it is crucial to ensure that these files are not overwritten when multiple calculations are executed simultaneously. To achieve this, you can create separate folders for each calculation and use a decorator that changes the working directory to the desired absolute path before executing the function.
Here's an example that demonstrates the use of a decorator to change the working directory and ensure that output files from different calculations do not overwrite each other:

import covalent as ct
from pathlib import Path


import os
from functools import wraps

def change_dir_and_execute(directory):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            current_dir = os.getcwd()
            try:
                os.chdir(directory)
                result = func(*args, **kwargs)
            finally:
                os.chdir(current_dir)
            return result
        return wrapper
    return decorator


path=Path(".").absolute()

@ct.electron
@change_dir_and_execute(path)
def job(val1, val2,file):
    with open(file, "w") as w:
        w.write(str(val1 + val2))
    return Path(".").absolute()

@ct.lattice
def workflow(file, val1, val2, val3, val4):
    job1 = job(val1, val2,file)
    return job1


file="example.txt"
dispatch_id = ct.dispatch(workflow)(file, 1, 2, 3, 4)
result = ct.get_result(dispatch_id, wait=True)
print(result)

In this example, the Electron is executed in the specified path (the current directory), ensuring that output files from different calculations are written to separate directories and not overwritten.

[Andrew-S-Rosen]

I marked this as the answer, but in my opinion this is a workaround and a long-term solution is to introduce some UX to support this. This is proposed in #1628.

[santoshkumarradha]

Wow many thanks for populating this ! had in my todo to pull out an discussion 🙏 !