Problems with auth when following redirects

[gardenia]

Hi,

I have written a proxy between the php and S3. One of the features of the proxy is to send redirects (status 307) back to the client so they can go direct and offload large payloads from the proxy. examples of which are,. GETs on large files or PUTs of multipart upload parts.

NOTE: for reasons I won't go into: I am using the "endpoint" feature to direct traffic to the proxy (rather than HTTP_PROXY). example:

<?php                                                                                                         
                                                                                                              
require "./aws-autoloader.php";                                                                               
                                                                                                              
use Aws\S3\S3Client;                                                                                          
use Aws\Exception\AwsException;                                                                               
                                                                                                              
$client = Aws\S3\S3Client::factory([                                                                          
    'profile' => 'foo',                                                                                       
    'version' => 'latest',                                                                                    
    'region' => 'eu-west-1',                                                                                  
    'debug' => true,                                                                                          
                                                                                                              
    'endpoint' => 'http://localhost:8081',                                                                    
    'use_path_style_endpoint' => true,                                                                        
]);                                                                                                           
                                                                                                              
$bucket = 'redacted-01';                                                                                         
$key = 'hello.txt';                                                                                           
                                                                                                              
try {                                                                                                         
    // Get the object.                                                                                        
    $result = $client->getObject([                                                                            
        'Bucket' => $bucket,                                                                                  
        'Key'    => $key,                                                                                     
    ]);                                                                                                       
                                                                                                              
    // Display the object in the browser.                                                                     
    echo $result['Body'];                                                                                     
} catch (S3Exception $e) {                                                                                    
    echo $e->getMessage() . PHP_EOL;                                                                          
}

The specific problem I'm having is that when the proxy returns a redirect (status 307) the client will misbehave in the following ways:

1] if the redirect is to a presigned URL then GuzzleHttp/RedirectMiddleware.php will retain the original request headers and try to resend them along with the presigned URL when following the redirect resulting in S3 complaining about extraneous request headers:

Error executing "GetObject" on "http://localhost:8081/redacted-01/hello.txt"; AWS HTTP error: Client error: GET http://localhost:8081/redacted-01/hello.txt resulted in a 403 Forbidden response:

AccessDeniedThere were headers present in the reques (truncated...)
AccessDenied (client): There were headers present in the request which were not signed -
AccessDeniedThere were headers present in the request which were not signedx-amz-dateD29503FF38483831k7QYPI0bjIf8iqOO0mgunxz9pYidg6IXZKbwqdtFNah3bqNT67cuDuptcOQFtjBSTAlDL2zeDPU=

I was able to do a local workaround for this as follows:

diff -Brub ../php_v3_ORIG/GuzzleHttp/RedirectMiddleware.php ./GuzzleHttp/RedirectMiddleware.php
--- ../php_v3_ORIG/GuzzleHttp/RedirectMiddleware.php	2019-04-05 19:11:38.000000000 +0100
+++ ./GuzzleHttp/RedirectMiddleware.php	2019-04-09 16:53:05.542379227 +0100
@@ -197,6 +197,10 @@
             $modify['remove_headers'][] = 'Authorization';
         }
 
+        $modify['remove_headers'][] = 'Referer';
+        $modify['remove_headers'][] = 'X-Amz-Content-Sha256';
+        $modify['remove_headers'][] = 'X-Amz-Date';
+
         return Psr7\modify_request($request, $modify);
     }
 

2] if the redirect is to a plain URL, e.g. :

< HTTP/1.1 307 Temporary Redirect
< Location: https://s3.eu-west-1.amazonaws.com/redacted-01/hello.txt
< Content-Length: 0

then there the same fix as above is required to slurp out the extraneous headers but additionally then the request gets sent without an "Authorisation" header. I believe that what needs to happen in this case is that the S3Client needs to calculate a new "Authorisation" header but am not familiar enough with the codebase to know how to get it to do that after the RedirectMiddleware. I'd be happy to have a stab at it if someone can give me some pointers.

NOTE: both of these cases work fine in boto.
Case #1: works fine in boto but NOT the S3 java SDK
Case #2: works fine in boto and the S3 java SDK

The PHP SDK (v3) doesn't work with either case.

Any feedback / suggestions, in particular how to workaround case #2 are very welcome.

Thanks.

[howardlopez]

For custom redirect behavior, I would first look and see how much you can accomplish with the http options in the AWS client, which are passed to Guzzle as request options. The allow_redirects option should give you a foothold to do what you need with the headers, as there is an on_redirect callback hook that might be helpful.

Note that if you are using Guzzle V5, the GuzzleHandler for V5 does have a whitelist for accepted options, which you'll need to add allow_redirects to.

[gardenia]

thanks. I will look into those pointers.

in case it wasn't clear originally, I didn't intend this as merely a "request for guidance" type ticket. I am arguing that both cases are potentily bugs in the PHP SDK. i realise it may be somewhat undefined area but it makes sense to me that if a redirect is sent by the server (or a middleman proxy) that the PHP SDK should follow that redirect and regenerate the auth header appropriately which is what boto and the java SDKs do in the same scenario.

[howardlopez]

Currently the redirect is handled completely by Guzzle before getting back to the SDK, and the behavior isn't currently customized. I don't think header removal on redirects is within the scope of the SDK - it assumes too much about the use case, and the user should have access to it with the callback option in allow_redirects as mentioned above. However, regenerating the authorization header for S3 redirects does make sense, and I will go ahead and mark this as a feature request. You are certainly welcome to tackle the PR if you like, and we'd be glad to give input when it's in progress.

[srchase]

@gardenia,

I've marked this a feature-request for the next major version of the SDK (v4).

As you point out, Guzzle removes the authorization header. Further, Guzzle'son_redirect callback is not able to modify the current or next request.

There is an entry point to re-sign requests using the current version of the SDK. A http_handler can be substituted for the default http handler. In providing a different http handler, you'd be able to modify the redirect behavior to enable re-signing redirects.