Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3g sidecar passthru #8287

Closed
wants to merge 51 commits into from
Closed

S3g sidecar passthru #8287

wants to merge 51 commits into from

Commits on Oct 5, 2022

  1. Switch to local s2 from external s2 module

    @msteffen @lukemarsden
    msteffen authored and lukemarsden committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    ed20c5c View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2022

  1. WIP

    @lukemarsden
    lukemarsden committed Oct 7, 2022
    Configuration menu
    Copy the full SHA
    49a8aef View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2022

  1. WIP on proxying to real S3/minio backend in certain cases for s3_out

    @lukemarsden
    lukemarsden committed Oct 11, 2022
    Configuration menu
    Copy the full SHA
    ee9cd86 View commit details
    Browse the repository at this point in the history

  2. WIP - getting a 404 for some reason. Need to understand the mux routi… 

    …ng s2 uses in more detail.
    @lukemarsden
    lukemarsden committed Oct 11, 2022
    Configuration menu
    Copy the full SHA
    9614e36 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2022

  1. More WIP

    @lukemarsden
    lukemarsden committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    5623711 View commit details
    Browse the repository at this point in the history

  2. example

    @lukemarsden
    lukemarsden committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    a906fe5 View commit details
    Browse the repository at this point in the history

  3. bump, debugging

    @lukemarsden
    lukemarsden committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    5f80fd0 View commit details
    Browse the repository at this point in the history

  4. debug logging from spark was making it unbearably slow

    @lukemarsden
    lukemarsden committed Oct 12, 2022
    Configuration menu
    Copy the full SHA
    1c75c42 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2022

  1. mega find and replace

    @lukemarsden
    lukemarsden committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    c2854f4 View commit details
    Browse the repository at this point in the history

  2. Debugging rewriting...

    @lukemarsden
    lukemarsden committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    c4447da View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2022

  1. try replacing bucket name and path separately

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    272da07 View commit details
    Browse the repository at this point in the history

  2. REVERSE the transformation in the response

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    ff66799 View commit details
    Browse the repository at this point in the history

  3. Handle XML style bucket rewriting too

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    d21527b View commit details
    Browse the repository at this point in the history

  4. Thank you linter (for once)

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    88ae87a View commit details
    Browse the repository at this point in the history

  5. why are you getting stuck

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    a863d24 View commit details
    Browse the repository at this point in the history

  6. Try NOT transforming the path within the bucket, see if that works

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    20892b9 View commit details
    Browse the repository at this point in the history

  7. Try to get back to sanity...

    @lukemarsden
    lukemarsden committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    de3068c View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2022

  1. When it works, debug logging in spark breaks loki

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    09f725d View commit details
    Browse the repository at this point in the history

  2. Switch back to trying to get bucket renaming working

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    1391f21 View commit details
    Browse the repository at this point in the history

  3. turn back on signing and start mutating bucket name again

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    ed532f7 View commit details
    Browse the repository at this point in the history

  4. The last commit worked! Now try manipulating the path as well

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    0158e0d View commit details
    Browse the repository at this point in the history

  5. add stack to log message. This is now working

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    0d2acb7 View commit details
    Browse the repository at this point in the history

  6. performance - don't rewrite octet streams

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    a6f9463 View commit details
    Browse the repository at this point in the history

  7. cleanup

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    00ffc99 View commit details
    Browse the repository at this point in the history

  8. more cleanup

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    075727d View commit details
    Browse the repository at this point in the history

  9. cleanup

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    94b6b57 View commit details
    Browse the repository at this point in the history

  10. revert vendoring s2, not needed for the direction we took

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    0e228bf View commit details
    Browse the repository at this point in the history

  11. Merge branch '2.3.x' into s3g-sidecar-passthru

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    e33bcfa View commit details
    Browse the repository at this point in the history

  12. Cleanup

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    e80fcc0 View commit details
    Browse the repository at this point in the history

  13. Some WIP towards supporting real S3 as well. Needs testing in a reali… 

    …stic environment though.
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    a2b4105 View commit details
    Browse the repository at this point in the history

  14. First pass on copying data from backend job-scoped bucket location to… 

    … pipeline output commit
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    e6c05b1 View commit details
    Browse the repository at this point in the history

  15. Try creating a new commit

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    fb8d917 View commit details
    Browse the repository at this point in the history

  16. Comment

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    60ebb8b View commit details
    Browse the repository at this point in the history

  17. Revert docker changes (#8293)

    @chainlink
    chainlink committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    846176d View commit details
    Browse the repository at this point in the history

  18. Put S3 copying code in the right place - it doesn't actually work yet… 

    … (see user container pipeline logs grep for PROXY)
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    0220cc8 View commit details
    Browse the repository at this point in the history

  19. Merge branch 's3g-sidecar-passthru' of github.com:pachyderm/pachyderm… 

    … into s3g-sidecar-passthru
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    13e5c88 View commit details
    Browse the repository at this point in the history

  20. The copy has to happen from the storage container because that has th… 

    …e storage secrets, and we don't want to leak them to the user code. The triggering and blocking on the copy however needs to happen from the user code. So we make an http call from the user container on localhost to a new API endpoint on the storage container called /finish, BEFORE we finish the commit in the worker, and block until it completes.
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    d53ea8b View commit details
    Browse the repository at this point in the history

  21. Revert "Revert docker changes (#8293)" 

    This reverts commit 846176d.
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    2296eb5 View commit details
    Browse the repository at this point in the history

  22. Improve logging

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    0f25c77 View commit details
    Browse the repository at this point in the history

  23. Revert "Revert "Revert docker changes (#8293)"" 

    This reverts commit 2296eb5.
    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    ceb09bb View commit details
    Browse the repository at this point in the history

  24. cleanup - this code wasn't meant to be here as well

    @lukemarsden
    lukemarsden committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    368a25c View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2022

  1. Attempt cleanup after copying job-scoped prefix into pfs

    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    d0392fc View commit details
    Browse the repository at this point in the history

  2. Revert "Revert "Revert "Revert docker changes (#8293)""" 

    This reverts commit ceb09bb.
    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    28b673a View commit details
    Browse the repository at this point in the history

  3. logging

    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    68b6cd2 View commit details
    Browse the repository at this point in the history

  4. Don't do the cleanup from inside the mfc

    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    31aa3d2 View commit details
    Browse the repository at this point in the history

  5. Revert "Revert "Revert "Revert "Revert docker changes (#8293)"""" 

    This reverts commit 28b673a.
    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    42f5351 View commit details
    Browse the repository at this point in the history

  6. Cleanup stale comments

    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    2420d4b View commit details
    Browse the repository at this point in the history

  7. More comment cleanup

    @lukemarsden
    lukemarsden committed Oct 19, 2022
    Configuration menu
    Copy the full SHA
    368583e View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2022

  1. Fix up bail out logic

    @chainlink
    chainlink committed Oct 20, 2022
    Configuration menu
    Copy the full SHA
    56c094c View commit details
    Browse the repository at this point in the history

  2. Fix up logic

    @chainlink
    chainlink committed Oct 20, 2022
    Configuration menu
    Copy the full SHA
    ead11f9 View commit details
    Browse the repository at this point in the history

  3. Update Sign4 call to use the Security Token

    @bbonenfant
    bbonenfant committed Oct 20, 2022
    Configuration menu
    Copy the full SHA
    af9704b View commit details
    Browse the repository at this point in the history