Enhance Velox AWS S3 client to use proxy configuration

[czentgr]

Hello,

currently, the AWS S3 client that is instantiated in the S3FileSystem.
In many cases programs wishing to access internet resources support a configuration for a proxy. Hence for a deployment of Prestissimo in an environment that requires usage of a proxy to access external resources this is beneficial.

The AWS client itself supports client configuration to use a proxy:
https://sdk.amazonaws.com/cpp/api/LATEST/root/html/md_docs_2_client_configuration___parameters.html

However, a design choice was made to NOT honor the OS environment variables (on Linux) that are available for a system configuration: aws/aws-sdk-cpp#1049.

Therefore, any such configuration must be made explicitly in Prestissimo (and hence Velox as it implements the S3FileSystem).

The proposal would be to

1. Look for the OS level environment configuration and use those settings and pass them through to the S3 client.
2. Add the proxy configuration at the hive catalog configuration level where the other S3 configuration occurs.

If we go with both options and both a re configured, the hive properties would override the OS level configuration.
OS level env variables would be HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

The hive configuration would likely affect Presto as well. But this is a separate discussion.

What are the thoughts of adding this to Velox?

Thanks!
Christian

[ashokku2022]

@mbasmanova @pedroerp waiting for your thoughts on this issue, any feedback would be highly appreciated, thanks!

[mbasmanova]

CC: @majetideepak

[majetideepak]

The hive configuration would likely affect Presto as well. But this is a separate discussion.

@czentgr, can you elaborate on this point further? I believe the end goal is for Presto to be able to talk to an S3-like service behind a proxy. Do we have a similar requirement on the Presto Java side? How does the proxy configuration work there?

[czentgr]

@majetideepak From my understanding it is already supported for Presto. The AWS Java client already uses the (HTTP/S_PROXY, NO_PROXY) environment variables - if configured. Therefore, no changes to the JVM options or configuration for Presto are required.

Please see the doc where it mentions the ClientConfiguration automatically reading the environment variables:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#getProxyHost

In addition, this is the code: https://github.com/aws/aws-sdk-java/pull/596/files

This is another difference between the AWS S3 Java client and C++ client. A user will have to explicitly set it in C++.

[czentgr]

As a result, Prestissimo should take a look at the environment variables and decide whether or not to set them for the client.
From a deployment perspective, setting the environment variables allows a common configuration in a host regardless of the worker type.

[majetideepak]

@czentgr thanks for the context. I feel we should implement 2) in Velox and 1) in Prestissimo. What do you think?

[czentgr]

Yes, I can see the value in doing it in different parts.
Basically Prestissimo would look at the env vars and the hive config and decide if it needs to add the proxy settings to the hive config (in memory) before the HiveConnectorFactory is registered.
Subsequently, Velox would look at the hive properties for the proxy and use them as needed when instantiating the S3 client.

[majetideepak]

Exactly!

[czentgr]

The environment variables (HTTP_PROXY and HTTPS_PROXY) provide a URI that contains the protocol, optionally a user and password, the IP of the proxy and the port for the proxy listener. For example, a configuration could look like this

https_proxy=http://user:pass@IP/HOST:PORT
http_proxy=http://user:pass@IP/HOST:PORT
no_proxy=.localhost,172.0.0.1

The user and password are optional.

To keep things simple, the hive config will contain the options such that it matches the S3 ClientConfiguration member.
That means, user, password, IP/host, port etc will have their own hive config option.
Prestissimo is responsible for parsing and extracting the relevant information.

S3 ClientConfiguration:

    Aws::Http::Scheme proxyScheme;
    Aws::String proxyHost;
    unsigned proxyPort;
    Aws::String proxyUserName;
    Aws::String proxyPassword;

New session properties:

hive_s3_http_proxy
hive_s3_https_proxy
hive_s3_no_proxy

Prestissimo will use the values from HTTP_PROXY or HTTPS_PROXY depending on the scheme configured by hive.s3.ssl.enabled.

Unlike the Java API, the C++ API does not allow to specify a no-proxy list. Instead, Prestissimo parses the content of NO_PROXY and checks if it matches the hive.s3.endpoint URL domain. If a match occurs, the proxy configuration is not set.

When a S3 client is instatiated in Velox.

[majetideepak]

I now wonder if we need hive properties since Java does not provide them.

[czentgr]

Well, these could be internal properties only to be used to exchange information. If they are present in a config they are ignored (aka removed/replaced in case the env vars are set by Prestissimo?)

[czentgr]

After working on the prototype some inconsistencies to the Java Presto behavior were noticed. Also having a fixed config would mean more changes in how the configs currently flow through Prestissimo and it was deemed to not match the Java behavior.

1. When the Java AWS client is instantiated the environment variables are read. Thus putting them into some config and pass down would harden them for the duration of the process. Instead, the S3FileSystem will read them when the S3 client is instantiated and evaluate them.
2. As a result of 1. we are going to introduce a new hive config variable to bypass looking at the proxy environment altogether. This is different than Java but that's because in Java the user has no choice. This allows us to keep the current Prestissimo behavior if so chosen by the config.
   The new hive config variable is

hive.s3.ignore-proxy

The default is false. If set to true the environment proxy configuration, if present, is evaluated.

[majetideepak]

This is completed here #8574