AWS: Make Fetches Concurrent

[ramonpetgrave64]

Title: Make Fetches Concurrent

Description:
Cartography up to now is entirely serial. But there are many opportunities for concurrency, particularly during the get_ phases of sync modules.

As a basic example: To get all the data needed for all AWS ECR Images, we:

1. call boto's method, to get a list of all repositories
2. for each repository, we call another boto method to list all images within that repository

The loop in the second step above is what we can make concurrent, saving time of network overhead when there are perhaps many thousands of repositories.

The boto clients are also thread-safe, meaning it's okay to run them in multiple concurrent threads.

Ideally, we would use the async version of client libraries, but boto does not support that. There is a project aiboto3 that seems to wrap around boto to make its methods async, but personally I would rather not.

I have a pending PR as another workaround #1192. The idea there is to do minimal refactoring by

1. converting our loops to async methods
2. calling boto's get methods from separate threads

To be more concrete with the ECR Images example: you might think to just convert get_ecr_repository_images to an async def, but that still does not yield any performance gain because inside of that is a long-running function call with boto's list_images method. Because of the GIL, it is ultimately the list_images method that needs to run in a separate thread, but I chose to keep the refactor simple by putting it's only caller in a separate thread.

I've read a bit about asyncio and GIL from the realpython guides, but I'm still new and would like to hear thoughts from the community.

Relevant Links:

• #1206: add async to aws:s3 and aws:ecr #1192
• https://realpython.com/python-gil/
• https://realpython.com/python-async-features/
• https://realpython.com/async-io-python/
• https://boto3.amazonaws.com/v1/documentation/api/1.19.0/guide/clients.html#multithreading-or-multiprocessing-with-clients

[achantavy]

Some things to consider:

How will we apply/refactor this change to all AWS modules?

Will it be easy for new module authors to copy this pattern in their own modules?

Can you describe the performance gain that you observed in #1192 with a bit more detail?

[ramonpetgrave64]

2. Not difficult. Mainly just converting loop bodies. The hardest part will be adding tests where they didn't exist before. But I think manual testing will be enough.

3. The s3 and ec3 syncs would take about 5 and 15 minutes, respectively. With the async, they now take about 35 seconds and 2 minutes, respectively.

4. It could also be applied to the other non-aws modules. But for aws, it will be a similar process, as in the PR. Here are the candidates I've identified, but we can open a new GH issue/project to track the effort:

   • dynamodb
   • ecs
   • eks
   • elasticsearch
   • emr
   • iam
     • This will be a difficult refactor
   • kms
   • lambda
   • resourcegroupstaggingapi
   • route53
   • sqs
   • ssm

[mpurusottamc]

We tried to incorporate multi-threading. One of the challenges we faced was around Rate Limits. boto3 has retry policies in place. But, it sometimes fails if there are a ton of concurrent requests going to AWS.

[ramonpetgrave64]

Yes, I've seen Throttling exceptions, but I've mitigated with the @backoff decorator with its expo waiter. Ratelimits vary by AWS service and each individual API, and they don't always publish those limits.

For example, I recently did some one-off async calls against iam's generate_service_last_accessed_details and get_service_last_accessed_details, where I did get throttling exceptons, but the @backoff decorator against that exception was enough, and I still got results in ~20% of the usual time without async.

Also, the default threadpool I'm proposing would only allow a maximum of 10 concurrent requests, though we could increase that for more gains.

[ramonpetgrave64]

Can you tell us about your own approach at multithreading?

[ramonpetgrave64]

@mpurusottamc I've just updated the PR to use @backoff builtin. Please take a look.