[BUG] For large DB, Metadata Prefetch sync status remains in "In Progress" indefinitely

[rjdp]

Describe the bug

We simmulated a large db by creating 200 schemas with 1500 tables each and 200 columns per table, when the database was synced it continues to stay in "In Progress" state, after 5-6 hrs we verified that 187 schemas and all its tables information were fetched and no progress was further made even though Datasource continued to be in "In Progress"

Explain the environment

• Chaos Genius version: release candidate 0.6.0
• OS Version / Instance: Ubuntu 18.04, AWS EC2
• Deployment type: Docker

Current behavior

• Configure a datasource with following details (internal to CG core team)

• and after adding when it starts syncing automatically and goes "In Progress" state even after a long time it doesnt change to "Live/Completed"

Expected behavior

• The prefetch task should not be left in an inconsistent state i.e status says "In Progress" but the task has probably died and not conitnuing to fetch remaining metadata . however large db and however long it might take to complete prefetch the the datasource should go into "Live/Completed" state

Screenshots

Additional context

• currently we fetch info about all schemas sequntially and for each table of schemas sequentially and this is bound to take lot of time
• as the queries are an I/O operation we can parallelise a batch of such queries to improve prefetch completion
• Also probably longer tasks timeout in celery (this is a speculation)

Logs

N/A