Cannot modify tasks from on-modify hooks

[lyndhurst]

Hi,

Context

I have taskwarrior 2.6.2, and tasklib 2.5.1-2 installed with pacman. I also have many hooks modifying or checking the validity of the added or modified task, they all work fine.
Since I am trying to implement some feature loosely resembling sub tasks, I need to 'cascade' changes made to "parent tasks" onto "children tasks".

Problem

I tried to reduce everything to the minimum to understand the errors I was getting, it turns out what works with on-add hooks do not in on-modify hooks. I cannot find any mention of that as a limitation in the docs. Maybe I missed something, in which case, sorry about that.

Question

Can I modify tasks in the taskwarrior database from on-modify hooks ? And, if yes, how ?

Steps to reproduce

Everything below is done with a 'minimal' config generated by taskwarrior, and no other hooks in place than the one being tested.

on-add Hook

#!/usr/bin/python3

import sys
from tasklib import TaskWarrior
from tasklib.task import Task

added_task = Task.from_input()

tw = TaskWarrior()

tw.execute_command(["status:pending", "+foo", "modify", "-foo", "+bar"],
                   config_override={'hooks': 'off'})

print(added_task.export_data())
sys.exit(0)

Works as expected:

❯ t

ID Age   Tag Description Urg 
 1 55min foo Some task    0.8
 2 54min foo Some task    0.8
 3 54min foo Some task    0.8

3 tasks
❯ t add Testing the hook
Created task 4.
❯ t

ID Age   Tag Description      Urg 
 1 55min bar Some task         0.8
 2 54min bar Some task         0.8
 3 54min bar Some task         0.8
 4  3s       Testing the hook    0

4 tasks

on-modify hook

Same hook as above expecting the same results:

#!/usr/bin/python3

import sys
from tasklib import TaskWarrior
from tasklib.task import Task

# Load the task attributes.
modified_task = Task.from_input()

tw = TaskWarrior()

tw.execute_command(["status:pending", "+foo", "modify", "-foo", "+bar"],
                   config_override={'hooks': 'off'})


print(modified_task.export_data())
sys.exit(0)

No error is thrown, but no effects on the database:

❯ t

ID Age   Tag Description      Urg 
 1  1h   foo Some task         0.8
 2 59min foo Some task         0.8
 3 59min foo Some task         0.8
 4  5min     Testing the hook    0

4 tasks
❯ t /Testing/ mod +test
Modifying task 4 'Testing the hook'.
Modified 1 task.
❯ t

ID Age  Tag  Description      Urg 
 1 1h   foo  Some task         0.8
 2 1h   foo  Some task         0.8
 3 1h   foo  Some task         0.8
 4 5min test Testing the hook  0.8

4 tasks

Additional observations

Modifying the on-modify hook so the initial command filtering does not return any matching tasks throws an error.

tw.execute_command(["status:pending", "+bar", "modify", "-bar", "+foo"],
                   config_override={'hooks': 'off'})

❯ t

ID Age   Tag  Description      Urg 
 1  1h   foo  Some task         0.8
 2  1h   foo  Some task         0.8
 3  1h   foo  Some task         0.8
 4 11min test Testing the hook  0.8

4 tasks
❯ t /Testing/ mod -test
Modifying task 4 'Testing the hook'.
Traceback (most recent call last):
  File "/home/user/.config/task/test/hooks/on-modify-tests", line 12, in <module>
    tw.execute_command(["status:pending", "+bar", "modify", "-bar", "+foo"],
  File "/usr/lib/python3.11/site-packages/tasklib/backends.py", line 279, in execute_command
    raise TaskWarriorException(error_msg)
tasklib.backends.TaskWarriorException: Configuration override rc.confirmation=no
Configuration override rc.dependency.confirmation=no
Configuration override rc.recurrence.confirmation=no
Configuration override rc.json.array=off
Configuration override rc.bulk=0
Configuration override rc.hooks=off
No tasks specified.
Command used: task rc.confirmation=no rc.dependency.confirmation=no rc.recurrence.confirmation=no rc.json.array=off rc.bulk=0 rc.hooks=off status:pending +bar modify -bar +foo
Hook Error: Expected feedback from failing hook script: on-modify-tests

Other attempts

Loop

Looping through tasks using tasklib produces exactly the same results as tw.execute_command. It also works fine from an on-add hook.

foo_tasks = tw.tasks.pending().filter("+foo")

tw.overrides.update({'hooks': 'off'})

for foo_task in foo_tasks:
    foo_task["tags"].remove("foo")
    foo_task["tags"].add("bar")
    foo_task.save()

tw.overrides.update({'hooks': 'off'})

subprocess produces interesting results

Trying to run the task command using the python subprocess module produces a taskwarrior as if it was modifying the tasks, but produces no effects on the database from an on-modify hook. Again, it works as expected from an on-add hook.

import subprocess

CMD = "task rc.hooks=off rc.confirmation=no"
CMD = CMD + " rc.dependency.confirmation=no rc.json.array=off"
CMD = CMD + " rc.recurrence.confirmation=no rc.bulk=0"
CMD = CMD + " status:pending +foo modify -foo +bar"

subprocess.run(CMD, shell=True, check=True)

Output from the on-modify hook:

❯ t

ID Age  Tag Description      Urg 
 1 1h   foo Some task         0.8
 2 1h   foo Some task         0.8
 3 1h   foo Some task         0.8
 4 4min     Testing the hook    0

4 tasks
❯ t /Testing/ mod +test
Modifying task 4 'Testing the hook'.
Configuration override rc.hooks=off
Configuration override rc.confirmation=no
Configuration override rc.dependency.confirmation=no
Configuration override rc.json.array=off
Configuration override rc.recurrence.confirmation=no
Configuration override rc.bulk=0
Modified 1 task.
This command will alter 3 tasks.
Modifying task 1 'Some task'.
Modifying task 2 'Some task'.
Modifying task 3 'Some task'.
Modified 3 tasks.
❯ t

ID Age  Tag  Description      Urg 
 1 1h   foo  Some task         0.8
 2 1h   foo  Some task         0.8
 3 1h   foo  Some task         0.8
 4 4min test Testing the hook  0.8

4 tasks

[smemsh]

Maybe modify rewrites the full serialized database, so you race with yourself in the hook. The db with your "new" changes (done behind back of hook-invoking taskwarrior) just gets overwritten when you return from hook. Whereas add might append, so your change survives. Just a guess after skimming your issue...

Looking at the code, indeed TF2::commit (done after dirty marked during update) says:

Special case: added but no modified means just append to the file

whereas modify case notes:

Truncate the file and rewrite

[lyndhurst]

Thanks for your input, I have been thinking of something like that even if I did not get as far as you in my understanding of the problem. Mostly because I do not navigate through the code base so easily...

I am using a dirty workaround that works surprisingly well for now. But I am thinking that if such a limitation was implemented on purpose by the developers, there must be a good reason for it and I might regret my workaround later down the road.

Thanks again for looking at it though 👍

[smemsh]

well it's just that each task instance assumes it is the only one running. it makes sense to disable writes from other instances during a hook, or serialize them with a lock, if you can fork off an instance to block on lock acquisition, but keep going in the parent, then the work will defer until the handler finishes.

once the deferred tasks acquired the lock they would need to re-read the database before doing any rewrite, as it may have been changed by the previous lock holder