Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Atomic writes to file #57

Open
tfiers opened this issue May 31, 2019 · 1 comment
Open

Atomic writes to file #57

tfiers opened this issue May 31, 2019 · 1 comment
Labels
performance Gotta go fast

Comments

@tfiers
Copy link
Owner

tfiers commented May 31, 2019

Each FileTarget must first write to a temp file, then when done, copy it out to final dest.

This is to avoid Luigi's atomic writes problem
(i.e. other workers thinking a task is complete when its output is still being written)

@tfiers tfiers added the performance Gotta go fast label May 31, 2019
@tfiers
Copy link
Owner Author

tfiers commented Jun 3, 2019

Cause of this?

Traceback for DownsampleRawRecording(config_id=dev, file_ID=Rec(rat 2, day 1, D15))
Runtime error:
Traceback (most recent call last):
  File "/export/opt/anaconda2/envs/tomas/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/export/opt/anaconda2/envs/tomas/lib/python3.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/mnt/nerfhf01/kloostermanwip/Tomas/code/sharp/sharp/tasks/base.py", line 45, in run
    self.work()
  File "/mnt/nerfhf01/kloostermanwip/Tomas/code/sharp/sharp/tasks/signal/downsample.py", line 57, in work
    self.output().write(Signal(signal_down, fs_new, "μV"))
  File "/mnt/nerfhf01/kloostermanwip/Tomas/code/sharp/sharp/data/files/numpy.py", line 57, in write
    dataset = f.create_dataset(self.KEY_SIG, data=signal.data)
  File "/export/opt/anaconda2/envs/tomas/lib/python3.7/site-packages/h5py/_hl/group.py", line 119, in create_dataset
    self[name] = dset
  File "/export/opt/anaconda2/envs/tomas/lib/python3.7/site-packages/h5py/_hl/group.py", line 287, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Gotta go fast
Projects
None yet
Development

No branches or pull requests

1 participant