Error #01: Needed resources are used by other tasks.

[ishaulov]

Hi! Need help with publishing packages!

Packages are assembled on Jenkins, and then they are further uploaded and published to the repository using the aptly api:

curl -f -X POST -F file=@"libson-1.11.6248_amd64.deb" http://repos.local/deb/api/files/uploaded-files?_async=true -u repouser:repopassword 
curl -f -X POST http://repos.local/deb/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./debrepo?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-20.04?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./mint?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-22.04??_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 

The problem is that there may be parallel builds, downloads, and publications at the same time, and I get the following error:
aptly — "Error #1: Needed resources are used by other tasks."
curl — (22) The requested URL returned error: 409 Conflict

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"info","remote":"172.27.0.4","method":"POST","path":"/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true","protocol":"HTTP/1.0","code":"202","latency":"166.954µs","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00"}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./debrepo?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.423243ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-20.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.386ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./mint?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.258301ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-22.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.272133ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

Have I tried using this version:
#1125 (comment)
and using ?_async=true, but the error persists.

I tried to build aptly from the master branch, errors persist.
What else can I do, or maybe I'm doing something wrong?

[neolynx]

Hi, thanks for reporting !

yes, this might be fixed with #1125, which is currently being rebased on master.

the ?_async=true will make the API calls return a task_id. the caller should wait for the task to complete, before issuing a next api call. this can be done with GET /tasks/:id/wait
maybe you could try this ?

[ishaulov]

Thanks for the reply!

Yes, I can. But let me clarify: do I understand correctly that I need to write a shell handler that will process each curl request, take the ID and put it into GET /tasks/:ID/wait ?

[neolynx]

yes, something like this:
curl http://repos.local/api/tasks/1/wait

[ishaulov]

Hello, @neolynx!

I apologize for the delay in replying, it took a while to check everything out.

I've added /tasks/:ID/wait in my shell script, but I still get this error when building in parallel on Jenkins:

[GIN] 2024/05/21 - 13:41:39 | 200 |   83.949261ms |      172.19.0.2 | GET      "/api/tasks/9/wait"
2024/05/21 13:41:39 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:39 | 202 |    2.403509ms |      172.19.0.2 | PUT      "/api/publish/:./ubuntu-20.04?_async=true"
[GIN] 2024/05/21 - 13:41:39 | 200 |   87.470772ms |      172.19.0.2 | GET      "/api/tasks/10/wait"
[GIN] 2024/05/21 - 13:41:47 | 200 |  310.719834ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 202 |      115.25µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6316_amd64.deb?_async=true"
[GIN] 2024/05/21 - 13:41:47 | 200 |  307.632802ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 409 |     116.276µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6317_amd64.deb?_async=true"
Error #01: Needed resources are used by other tasks.

Aptly version 1.5.0+ds1-1+b4 .

This is stdout of shell script with set -x:

+ echo 'Moving deb package to repository...'
Moving deb package to repository...
++ curl -f -X POST 'http://repos.local/deb/api/repos/libs/file/uploaded-files//libs-generator.87.6317_amd64.deb?_async=true' -u repo-api:aGF#EQLr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    53    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 409

And do I understand correctly that I can't get ID and STATE on this request?

+ echo 'Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb'
Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb
++ curl -s -f -X POST -F file=@libsdeb///libs-generator.87.6317_amd64.deb 'http://repos.local/deb/api/files/uploaded-files?_async=true'
+ upload_response='["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ '[' 0 -ne 0 ']'
+ echo '["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ json_pp
[
   "uploaded-files//libs-generator.87.6317_amd64.deb"
]

And is it correct that on GET /tasks/:ID/wait I should get STATE=2? Is there any actual documentation on this function somewhere?