Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Secure Tunnel Connection Using Username and Password Fails from the AWS Console #152

Open
Adesanya-Toba opened this issue May 8, 2024 · 7 comments
Open

Secure Tunnel Connection Using Username and Password Fails from the AWS Console #152

Adesanya-Toba opened this issue May 8, 2024 · 7 comments
Labels
bug Something isn't working

Comments

@Adesanya-Toba
Copy link

Adesanya-Toba commented May 8, 2024 •
edited

Describe the bug
After starting the local proxy on my destination device, I attempt to open an SSH connection to the device using the username and password field from the AWS console. This results in an error in the local proxy running on the destination device, causing the device to disconnect.

To Reproduce

  1. Create a new tunnel from the AWS console
  2. Obtain the destination access token and start the local proxy on the destination device. I use a config file to store credentials.
  3. Attempt to connect from the AWS console using username and password.

Expected behavior

  • A browser-based shell is opened and I have access to the device.

Actual behavior

  • Browser-based shell fails to open and the local proxy on the device experiences an error and fails with the following logs (in verbose mode).

Logs

root@iot-gate-imx8plus:/home/dependencies/aws-iot-securetunneling-localproxy/build/bin# ./localproxy --config /home/local_proxy/config.ini -v 6
[2024-05-08 12:11:45.565299] (0x0000ffffa9a24000) [debug] Detect port mapping configuration provided through CLI in destination mode:
[2024-05-08 12:11:45.565450] (0x0000ffffa9a24000) [debug] ----------------------------------------------------------
[2024-05-08 12:11:45.565491] (0x0000ffffa9a24000) [debug] SSH = 22
[2024-05-08 12:11:45.565525] (0x0000ffffa9a24000) [debug] ----------------------------------------------------------
[2024-05-08 12:11:45.565619] (0x0000ffffa9a24000) [debug] /home/dependencies/aws-iot-securetunneling-localproxy/build/bin/config does not exist!
[2024-05-08 12:11:45.565747] (0x0000ffffa9a24000) [info] Starting proxy in destination mode
[2024-05-08 12:11:45.565803] (0x0000ffffa9a24000) [trace] Setting up web socket...
[2024-05-08 12:11:45.575563] (0x0000ffffa9a24000) [trace] Calling control_callback with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.575760] (0x0000ffffa9a24000) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:11:45.575787] (0x0000ffffa9a24000) [trace] Resolving proxy server host: data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:11:45.577803] (0x0000ffffa9a24000) [debug] Resolved proxy server IP: 18.192.167.159
[2024-05-08 12:11:45.577899] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.605751] (0x0000ffffa9a24000) [debug] Connected successfully with proxy server
[2024-05-08 12:11:45.605872] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.605940] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.606070] (0x0000ffffa9a24000) [trace] Performing SSL handshake with proxy server
[2024-05-08 12:11:45.606114] (0x0000ffffa9a24000) [trace] Calling set_verify_mode with type: single_ssl_stream
[2024-05-08 12:11:45.606195] (0x0000ffffa9a24000) [trace] Calling set_verify_callback with type: single_ssl_stream
[2024-05-08 12:11:45.606247] (0x0000ffffa9a24000) [trace] Calling next_layer().async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.606292] (0x0000ffffa9a24000) [trace] SSL next_layer() SNI is set : data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:11:45.644859] (0x0000ffffa9a24000) [debug] Successfully completed SSL handshake with proxy server
[2024-05-08 12:11:45.645008] (0x0000ffffa9a24000) [trace] Performing websocket handshake with proxy server
[2024-05-08 12:11:45.645088] (0x0000ffffa9a24000) [trace] Calling async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.645335] (0x0000ffffa9a24000) [trace] Web socket ugprade request(*not entirely final):
GET /tunnel?local-proxy-mode=destination HTTP/1.1
Host: data.tunneling.iot.eu-central-1.amazonaws.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: XK9zgz2STllsD3Kvh1eJAw==
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: aws.iot.securetunneling-3.0
access-token: ***ACCESS_TOKEN_REMOVED***
User-Agent: localproxy linux 64-bit/boost-1.84.0/openssl-3.0.0/protobuf-3.17.3

[2024-05-08 12:11:45.731195] (0x0000ffffa9a24000) [trace] Web socket upgrade response:
HTTP/1.1 101 Switching Protocols
Date: Wed, 08 May 2024 12:11:45 GMT
Content-Length: 0
Connection: upgrade
channel-id: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
upgrade: websocket
sec-websocket-accept: zGXDkON73pbICVGndgm6tOfJ46g=
sec-websocket-protocol: aws.iot.securetunneling-3.0

[2024-05-08 12:11:45.731637] (0x0000ffffa9a24000) [trace] Calling binary with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.731684] (0x0000ffffa9a24000) [trace] Calling auto_fragment with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.731711] (0x0000ffffa9a24000) [info] Web socket session ID: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
[2024-05-08 12:11:45.731742] (0x0000ffffa9a24000) [debug] Web socket subprotocol selected: aws.iot.securetunneling-3.0
[2024-05-08 12:11:45.731768] (0x0000ffffa9a24000) [info] Successfully established websocket connection with proxy server: wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:11:45.731834] (0x0000ffffa9a24000) [debug] Seting up web socket pings for every 20000 milliseconds
[2024-05-08 12:11:45.731863] (0x0000ffffa9a24000) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732061] (0x0000ffffa9a24000) [trace] Waiting for service ids...
[2024-05-08 12:11:45.732119] (0x0000ffffa9a24000) [trace] async_web_socket_read_loop_for_service_ids
[2024-05-08 12:11:45.732145] (0x0000ffffa9a24000) [debug] Scheduled next read:
[2024-05-08 12:11:45.732168] (0x0000ffffa9a24000) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732472] (0x0000ffffa9a24000) [trace] Processing control message
[2024-05-08 12:11:45.732565] (0x0000ffffa9a24000) [trace] Using global control message handler
[2024-05-08 12:11:45.732596] (0x0000ffffa9a24000) [debug] Extracting service Ids from control message 5
[2024-05-08 12:11:45.732629] (0x0000ffffa9a24000) [trace] Service id received: 
[2024-05-08 12:11:45.732653] (0x0000ffffa9a24000) [trace] SSH
[2024-05-08 12:11:45.732679] (0x0000ffffa9a24000) [trace] Validating service ids configuration
[2024-05-08 12:11:45.732706] (0x0000ffffa9a24000) [trace] Setting up tcp sockets 
[2024-05-08 12:11:45.732729] (0x0000ffffa9a24000) [trace] Clearing all ws data buffers
[2024-05-08 12:11:45.732751] (0x0000ffffa9a24000) [trace] Finished Clearing all ws data buffers
[2024-05-08 12:11:45.732774] (0x0000ffffa9a24000) [trace] Initializing tcp clients ...
[2024-05-08 12:11:45.732815] (0x0000ffffa9a24000) [trace] Setting up tcp socket for service id: SSH
[2024-05-08 12:11:45.732845] (0x0000ffffa9a24000) [trace] Waiting for stream start...
[2024-05-08 12:11:45.732870] (0x0000ffffa9a24000) [debug] Starting web socket read loop continue reading...
[2024-05-08 12:11:45.732893] (0x0000ffffa9a24000) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732972] (0x0000ffffa9a24000) [trace] return continue_reading true
[2024-05-08 12:11:45.733013] (0x0000ffffa9a24000) [debug] Starting web socket read loop while web socket is already reading. Ignoring...
[2024-05-08 12:12:05.732449] (0x0000ffffa9a24000) [trace] Sent ping data: 1715170325732
[2024-05-08 12:12:05.732546] (0x0000ffffa9a24000) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.251923] (0x0000ffffa9a24000) [trace] Pong reply latency: 26520 ms
[2024-05-08 12:12:12.252044] (0x0000ffffa9a24000) [trace] Pong reply latency: 6520 ms
[2024-05-08 12:12:12.281726] (0x0000ffffa9a24000) [trace] Processing control message
[2024-05-08 12:12:12.281827] (0x0000ffffa9a24000) [trace] Using global control message handler
--------------------------------- ERROR HERE -------------------------------------------------
[2024-05-08 12:12:12.282461] (0x0000ffffa9a24000) [error] Error from io_ctx::run(): Receive stream start before receiving service ids. Cannot forward data.
[2024-05-08 12:12:12.282553] (0x0000ffffa9a24000) [error] Failed web socket session ID: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
--------------------------------- ERROR HERE -------------------------------------------------
[2024-05-08 12:12:12.283583] (0x0000ffffa9a24000) [trace] Setting up web socket...
[2024-05-08 12:12:12.285685] (0x0000ffffa9a24000) [trace] Calling control_callback with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.285806] (0x0000ffffa9a24000) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:12:12.285856] (0x0000ffffa9a24000) [trace] Resolving proxy server host: data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:12:12.329890] (0x0000ffffa9a24000) [debug] Resolved proxy server IP: 35.157.95.122
[2024-05-08 12:12:12.330002] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356690] (0x0000ffffa9a24000) [debug] Connected successfully with proxy server
[2024-05-08 12:12:12.356806] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356850] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356888] (0x0000ffffa9a24000) [trace] Performing SSL handshake with proxy server
[2024-05-08 12:12:12.356920] (0x0000ffffa9a24000) [trace] Calling set_verify_mode with type: single_ssl_stream
[2024-05-08 12:12:12.356953] (0x0000ffffa9a24000) [trace] Calling set_verify_callback with type: single_ssl_stream
[2024-05-08 12:12:12.356993] (0x0000ffffa9a24000) [trace] Calling next_layer().async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.357030] (0x0000ffffa9a24000) [trace] SSL next_layer() SNI is set : data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:12:12.395437] (0x0000ffffa9a24000) [debug] Successfully completed SSL handshake with proxy server
[2024-05-08 12:12:12.395525] (0x0000ffffa9a24000) [trace] Performing websocket handshake with proxy server
[2024-05-08 12:12:12.395570] (0x0000ffffa9a24000) [trace] Calling async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.395615] (0x0000ffffa9a24000) [trace] Web socket ugprade request(*not entirely final):
GET /tunnel?local-proxy-mode=destination HTTP/1.1
Host: data.tunneling.iot.eu-central-1.amazonaws.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: vp9sMsZfS4kzKMjRbwuuig==
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: aws.iot.securetunneling-3.0
access-token: ***ACCESS_TOKEN_REMOVED***
User-Agent: localproxy linux 64-bit/boost-1.84.0/openssl-3.0.0/protobuf-3.17.3

[2024-05-08 12:12:12.479306] (0x0000ffffa9a24000) [trace] Web socket upgrade response:
HTTP/1.1 400 Bad Request
Date: Wed, 08 May 2024 12:12:12 GMT
Content-Type: text/plain
Content-Length: 85
Connection: keep-alive
channel-id: 029836fffe37c32d-00001778-0005a26f-dd8f498194830f15-5371459f
X-Status-Reason: Invalid access token: The access token was previously used and can not be used again.

Invalid access token: The access token was previously used and can not be used again.
[2024-05-08 12:12:12.479451] (0x0000ffffa9a24000) [error] Proxy server rejected web socket upgrade request: (HTTP/1.1 400 Bad Request) "Invalid access token: The access token was previously used and can not be used again."
[2024-05-08 12:12:12.479528] (0x0000ffffa9a24000) [trace] Calling is_open with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.479565] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.479600] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type

AWS Console Screenshot
Error appears as soon as I click connect from the Cosole.
image

Console connection fails (Connect button is greyed out) and requires a token rotation.
image

Environment (please complete the following information):

  • OS: Debian
  • Version Linux 5.15.32+g9d7d040bf936
  • Architecture: arm64 (aarch64 GNU/Linux)
  • Localproxy commit: public.ecr.aws/aws-iot-securetunneling-localproxy/debian-base:arm64-latest

Additional context
NOTE: I am able to successfully connect when using the local proxy on my machine as the source device! The failure only happens when I try to connect via the AWS console.

I pulled the latest Debian base image and built the local proxy inside Docker for my platform.
I ran it with the binary image and obtained the same error.
@Adesanya-Toba Adesanya-Toba added the bug Something isn't working label May 8, 2024
@Adesanya-Toba
Copy link
Author

Hi all, any thoughts on this, as I think it might be a small thing I'm missing?

As I said, it works when I use my local machine (i.e., laptop running the local proxy) as the source device but fails when I use the AWS console as the source device.

@RogerZhongAWS
Copy link
Contributor

Hey @Adesanya-Toba thanks for opening this issue. I was able to quickly reproduce the same error you are seeing. Do you know if this happening as a result of a recent update? (if you used a previous version of the localproxy, was it working before?)

@Adesanya-Toba
Copy link
Author

Hi @RogerZhongAWS, thanks for checking this out. No, I haven't tested earlier versions of the local proxy.

@RogerZhongAWS
Copy link
Contributor

@Adesanya-Toba , In a recent update, we actually added a new CLI option that the user needs to pass when connecting the localproxy to tunnel clients that only support older versions of the protocol. Because the console uses an older version of the protocol (V1) you will need to pass in --destination-client-type V1 into your localproxy run command (or add it within your config)

I also realize that the existing documentation may be a little confusing to understand, will take a moment to fix that.

@Adesanya-Toba
Copy link
Author

Oh, alright. Thanks, I'll give this a spin and feedback.

@Adesanya-Toba
Copy link
Author

Hi @RogerZhongAWS, yeah that did the trick! 🎉

Any plans to update the local proxy protocol on the console though?

@RogerZhongAWS
Copy link
Contributor

Can't answer that yet by myself, will need to consult with various people internally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Adesanya-Toba @RogerZhongAWS