Check the number of task in the queue

[enzezhang]

I am a user of gee_tools. I usually export image collection to google drive. However, when the number of the task in the queue is large than 3000, the program will be halted. I wonder if there is any way to check the number of the task in the queue and pause the program when the number is large?

Thanks for your kind help and looking forward to your reply.

Best,
Enze

[fitoprincipe]

Hi @enzezhang ! Yes, that is a limit imposed by GEE. I think it's a good idea, I'd need to think how to implement it. Thanks =)

[enzezhang]

Hi @fitoprincipe, I have made some changes to imagecollection.py. The mean purpose is to check the status of the last task every 10 seconds (it could be longer) after a series of tasks were submitted. When the last task becomes "COMPLETED", the program will keep moving.

after line 79, I add:
if n%20==0 and n>19:
while True:
ID=tasklist[n-2].id
status=ee.data.getTaskStatus(ID)
state=status[0].get('state')
if state=="COMPLETED":
break
else:
print("waiting, check on %s"%time.ctime())
time.sleep(10)

[enzezhang]

Hope my changes would be useful to you and others.

[aadityaverma]

Hi @fitoprincipe, I have made some changes to imagecollection.py. The mean purpose is to check the status of the last task every 10 seconds (it could be longer) after a series of tasks were submitted. When the last task becomes "COMPLETED", the program will keep moving.

after line 79, I add: if n%20==0 and n>19: while True: ID=tasklist[n-2].id status=ee.data.getTaskStatus(ID) state=status[0].get('state') if state=="COMPLETED": break else: print("waiting, check on %s"%time.ctime()) time.sleep(10)

whats the corrent indentation for the above code snippet and what is the assigned value for variable 'n ' ?

[enzezhang]

Hi All, I attached the code. The n is the current number of pictures when looping through the list. Best, Enze On May 27, 2022 09:49，Aditya ***@***.***> wrote： Hi @fitoprincipe, I have made some changes to imagecollection.py. The mean purpose is to check the status of the last task every 10 seconds (it could be longer) after a series of tasks were submitted. When the last task becomes "COMPLETED", the program will keep moving. after line 79, I add: if n%20==0 and n>19: while True: ID=tasklist[n-2].id status=ee.data.getTaskStatus(ID) state=status[0].get('state') if state=="COMPLETED": break else: print("waiting, check on %s"%time.ctime()) time.sleep(10) whats the corrent indentation for the above code snippet and what is the assigned value for variable 'n ' ? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fitoprincipe]

Hi @fitoprincipe, I have made some changes to imagecollection.py. The mean purpose is to check the status of the last task every 10 seconds (it could be longer) after a series of tasks were submitted. When the last task becomes "COMPLETED", the program will keep moving.

after line 79, I add: if n%20==0 and n>19: while True: ID=tasklist[n-2].id status=ee.data.getTaskStatus(ID) state=status[0].get('state') if state=="COMPLETED": break else: print("waiting, check on %s"%time.ctime()) time.sleep(10)

Hi @enzezhang , is this in a pull requests??

[enzezhang]

It is not in a pull request. Best, Enze On May 27, 2022 10:15，Rodrigo E. ***@***.***> wrote： Hi @fitoprincipe, I have made some changes to imagecollection.py. The mean purpose is to check the status of the last task every 10 seconds (it could be longer) after a series of tasks were submitted. When the last task becomes "COMPLETED", the program will keep moving. after line 79, I add: if n%20==0 and n>19: while True: ID=tasklist[n-2].id status=ee.data.getTaskStatus(ID) state=status[0].get('state') if state=="COMPLETED": break else: print("waiting, check on %s"%time.ctime()) time.sleep(10) Hi @enzezhang , is this in a pull requests?? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fitoprincipe]

what if there are other tasks running??? I think the code should cover that...

[enzezhang]

There will be a lot of tasks running at the same time, but the number of tasks that are running cannot be too large. Therefore, I set a while loop to check the states the last task that is running currently. If that task is finished, the code will move on to the rest of the list. Otherwise, the code will be stopped until that task is finished. The code will not touch any job that is running, it will only check the status of the task and tell the code to wait. The break is only for the while loop that checks the job status, it will not stop the entire code.

[enzezhang]

I've used this code for a while and it works fine.

[enzezhang]

Also, the number 20 can be changed to any number as long as the number is not close to 3000.

[enzezhang]

“Otherwise, the code will be stopped until that task is finished.” By "stopped", I mean waiting.

[fitoprincipe]

This would be my approach: 3d67fb5. I tested it and works fine

[enzezhang]

Great! On May 27, 2022 12:27，Rodrigo E. ***@***.***> wrote： This would be my approach: 3d67fb5. I tested it and works fine — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fitoprincipe]

You can test it and see if it works as expected.. if it does I can populate the code to the other functions and finally merge and release..

[enzezhang]

I look at it again, and I think it should be “i<=secure” but not “i+n<=secure”. i is the number of tasks in queue, and n is the number of current task. Best, Enze On May 27, 2022 13:02，Rodrigo E. ***@***.***> wrote： You can test it and see if it works as expected.. if does I can populate the code the the other functions and finally merge and release.. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fitoprincipe]

You are right @enzezhang. I've already modified it, you can check it now. I have migrated the functions to utils so I can use them in the other functions. Thanks =)

[enzezhang]

Hi, The data exporting is controlled by the command "task.start()”, in line 65 of imagecollection.py. So the status checking has to be after that command and in the same loop. In the current version, the status checking is after the entire data exporting loop. So the status checking will not be triggered if the program reaches the 3000 limit. Thanks for your contribution. Best, Enze On May 27, 2022 17:41，Rodrigo E. ***@***.***> wrote： You are right @enzezhang. I've already modified it, you can check it now. I have migrated the functions to utils so I can use them in the other functions. Thanks =) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fitoprincipe]

🤔 I've created a mock for testing: https://gist.github.com/fitoprincipe/09d4c24050cd971be475b41790a3f462
Have a look and let me know. I've added some comments to understand the flow. I think it works as expected.
You can also test it here: https://replit.com/@RodrigoPrincipe/exportimagecollectionmock#main.py

[enzezhang]

I believe this one can work. Best, Enze On May 28, 2022 13:30，Rodrigo E. ***@***.***> wrote： 🤔 I've created a mock for testing: https://gist.github.com/fitoprincipe/09d4c24050cd971be475b41790a3f462 Have a look and let me know. I've added some comments to understand the flow. I think it works as expected. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>