Issue with pika.connection.update_secret

[rshandilya01]

I have an AWS Lambda rotator to rotate secret on RabbitMQ server running on Amazon MQ. Rotator has multiple steps including set and test secrets. I am using pika connection method - update_secret - to update the connection. This stage goes fine with no errors thrown. When testing secret with the new password, it throws an error. I am wondering if update_password is working correctly. I am rotating the password for the admin user.

set_secret code -
try:
conn.update_secret(pending_dict['password'], 'set_secret AWS Secret Rotation by Lambda')
except pika.exceptions.ConnectionWrongStateError as connectionerror:
logger.error(
"set_secret: ConnectionWrongStateError while updating Secret on RabbitMQ Broker: Connection is not open or available",
connectionerror)
except Exception as error:
logger.error("set_secret: Error when updating Secret on RabbitMQ Broker:", error)
finally:
conn.close()
logger.info("set_secret: Successfully set secret for %s." % arn)

Log statements -
2023-11-16T11:39:50.841-06:00

Copy
[INFO] 2023-11-16T17:39:50.841Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Closing transport socket and unlinking: state=3; <ssl.SSLSocket fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('169.254.76.1', 46510), raddr=('10.191.69.188', 5671)>
[INFO] 2023-11-16T17:39:50.841Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Closing transport socket and unlinking: state=3; <ssl.SSLSocket fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('169.254.76.1', 46510), raddr=('10.191.69.188', 5671)>

2023-11-16T11:39:50.842-06:00

Copy
[ERROR] 2023-11-16T17:39:50.842Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - None
[ERROR] 2023-11-16T17:39:50.842Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - None

2023-11-16T11:39:50.859-06:00

Copy
[ERROR] 2023-11-16T17:39:50.842Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Error in _create_connection().
Traceback (most recent call last):
File "/opt/python/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'
[ERROR] 2023-11-16T17:39:50.842Z 8fda8aeb-3351-4569-b44b-2fc419603c2d Error in _create_connection(). Traceback (most recent call last): File "/opt/python/pika/adapters/blocking_connection.py", line 451, in _create_connection raise self._reap_last_connection_workflow_error(error) pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'

2023-11-16T11:39:50.859-06:00

Copy
Traceback (most recent call last):
Traceback (most recent call last):

2023-11-16T11:39:50.859-06:00

Copy
File "/var/task/lambda_function.py", line 294, in get_mq_connection
File "/var/task/lambda_function.py", line 294, in get_mq_connection

2023-11-16T11:39:50.859-06:00

Copy
connection = pika.BlockingConnection(parameters)
connection = pika.BlockingConnection(parameters)

2023-11-16T11:39:50.859-06:00

Copy
File "/opt/python/pika/adapters/blocking_connection.py", line 360, in init
File "/opt/python/pika/adapters/blocking_connection.py", line 360, in init

2023-11-16T11:39:50.859-06:00

Copy
self._impl = self._create_connection(parameters, _impl_class)
self._impl = self._create_connection(parameters, _impl_class)

2023-11-16T11:39:50.859-06:00

Copy
File "/opt/python/pika/adapters/blocking_connection.py", line 451, in _create_connection
File "/opt/python/pika/adapters/blocking_connection.py", line 451, in _create_connection

2023-11-16T11:39:50.859-06:00

Copy
raise self._reap_last_connection_workflow_error(error)
raise self._reap_last_connection_workflow_error(error)

2023-11-16T11:39:50.859-06:00

Copy
pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'
pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'

[lukebakken]

What versions of the following software are you using:

• RabbitMQ and Erlang
• Python
• Pika

When testing secret with the new password, it throws an error

Are you certain the new secret is available to RabbitMQ?

What does RabbitMQ log at the same time your application logs the ProbableAuthenticationError error?

[rshandilya01]

Luke - here is my set_secret method - it is creating a new connection using current secret and then updating the same with the new secret. Is this what you are asking for?

I am able to login into the Management UI with the original password that I set up when creating the broker, but not with the new secret, which indicated to me that update secret did not work.

try:
# Secrets Manager automatically adds the AWSPREVIOUS staging label to the previous version,
# so that you retain the last known good version
previous_dict = get_secret_dict(service_client, arn, "AWSPREVIOUS")
except (service_client.exceptions.ResourceNotFoundException, KeyError):
previous_dict = None

current_dict = get_secret_dict(service_client, arn, "AWSCURRENT")
pending_dict = get_secret_dict(service_client, arn, "AWSPENDING", token)

# Now try the current password
conn = get_mq_connection(current_dict, "set_secret")
if not conn:
    if previous_dict:
        # If current does not work, try previous.
        conn = get_mq_connection(previous_dict, "set_secret")

# If we still don't have a connection, raise a ValueError
if not conn:
    logger.error("set_secret: Unable to log into database with previous or current secret of secret arn %s" % arn)
    raise ValueError("set_secret: Unable to log into database with previous or current secret of secret arn %s" % arn)

"""Update MQ broker with new password"""
try:
    logger.info("set_secret: Updating Secret in the MQ Broker Connection Adapter")
    conn.update_secret(pending_dict['password'], 'set_secret AWS Secret Rotation by Lambda')
    logger.info("set_secret: Updated Secret in the MQ Broker Connection Adapter")
except pika.exceptions.ConnectionWrongStateError as connectionerror:
    logger.error(
        "set_secret: ConnectionWrongStateError while updating Secret on RabbitMQ Broker: Connection is not open or available",
        connectionerror)
except Exception as error:
    logger.error("set_secret: Error when updating Secret on RabbitMQ Broker:", error)
finally:
    conn.close()
logger.info("set_secret: Successfully set secret for %s." % arn)

[lukebakken]

https://www.rabbitmq.com/oauth2.html#token-expiration

update_secret will only update the secret for the current connection (not globally). New connections, or logging into the Management UI, must acquire the new secret before accessing RabbitMQ.

My guess at this point is that you do not have RabbitMQ set up correctly to acquire the new secret.

I doubt I will be able to assist you further unless you can provide an environment that reproduces this issue. docker compose is a good option.

@MarcialRosales - is my understanding correct?

[rshandilya01]

Not sure what you meant above - use case is pretty simple. Please let me explain again - I am rotating password for the admin user. I am using old password (say, x) to connect to the server with admin user. I am using the same connection to update new password (say, y) and then close the connection. When I am trying to connect to the server again (this time with password as y and user as admin), I am getting the above error. When I try to login to the Management UI with password x and user as admin, I am able to. That indicates to me that update_secret did not work as expected, hence the discussion.

[lukebakken]

The update_secret method DOES NOT update the password for a user in RabbitMQ.

Please carefully read the docs to which I linked. Updating secrets is generally used in environments that use OAuth2 authentication.

If you are not using OAuth2, your "AWS Lambda rotator" process will have to update the password for the admin user (or whatever user you're using) by using either the rabbitmqctl command line, or by using the HTTP API.

To update a user's password, you would make a POST request to /api/users/USERNAME with the correct payload.

[rshandilya01]

Thanks Luke - that answers my question. I will use the HTTP API.

[MarcialRosales]

That is correct @lukebakken . This method is only meant to "re-authenticate" a live connection, i.e. to change the credentials which have already changed on the backend. This is not meant to change the credentials on the authentication backend.
If the password has changed in the backend, while the user is connected, the next time the user performs any action over the connection is going to be rejected unless the user has "updated" the password on the live connection.