Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiPartUpload support #41

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

MultiPartUpload support #41

wants to merge 11 commits into from

Conversation

MrPigss
Copy link

@MrPigss MrPigss commented Feb 16, 2023

While this PR should be working but is not yet ready to be merged.
No tests have been written. The functionality has only been tested using a Minio s3 bucket.

Any remarks / tips are welcome.
Help with mocking the request for testing would be appreciated.

This should implement following api's:

Todo:

  • Fix AWSv4Auth compatibility -> only public buckets work for now
  • write mocks
  • write tests

Example

import asyncio
from aioaws.s3 import S3Client, S3Config
from httpx import AsyncClient


async def s3_demo(client: AsyncClient):
    s3 = S3Client(client, S3Config('<access key>', '<secret key>', '<region>', 'my_bucket_name.com'))

    await s3.delete('path/testt.txt')

    async with s3.createMultipartUpload('path/testt.txt') as mpu:
       
        # These can also be uploaded in parallel using tasks
        await mpu.uploadPart(1, 'abcde' * 1_000_000)
        await mpu.uploadPart(2, 'abcde' * 1_000_000)
        await mpu.uploadPart(2, 'end')  # last part doesn't have to be 5mb

        x = await mpu.listParts(1000)
        print(x)  # should print 3 parts

    async for x in s3.list():
        print(x)  # should contain testt.txt


async def main():
    async with AsyncClient(timeout=30) as client:
        await s3_demo(client)


asyncio.run(main())

@codecov
Copy link

codecov bot commented Feb 16, 2023

Codecov Report

Merging #41 (316909d) into main (332855f) will decrease coverage by 7.25%.
The diff coverage is 24.19%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #41      +/-   ##
==========================================
- Coverage   96.64%   89.39%   -7.25%     
==========================================
  Files          10       10              
  Lines         626      679      +53     
  Branches      102      114      +12     
==========================================
+ Hits          605      607       +2     
- Misses         15       64      +49     
- Partials        6        8       +2     
Impacted Files Coverage Δ
aioaws/ses.py 95.73% <ø> (ø)
aioaws/s3.py 72.37% <21.66%> (-26.83%) ⬇️
aioaws/core.py 98.34% <100.00%> (-1.66%) ⬇️
aioaws/testing.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 332855f...316909d. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant