FAM frontend 403 error

[MCatherine1994]

Describe the task
Craig found an issue that he's trying to add a few roles assignments and suddenly he can't do anything anymore, he still seems "logged in" FAM, but can't perform any action. Refresh doesn't help.

Acceptance Criteria

• Investigate the 403 error
• second
• third

Additional context

• The access token should expire in 5min, but it should auto refresh if the user refresh the page or go to a new route

[MCatherine1994]

Checked cloud watch API Lambda log, after user login 5 mins, backend can't find the requester by cognito_user_id anymore...

[ianliuwk1019]

Hi @MCatherine1994 , I am looking at SqlAlchemy doc for "engines".
Perhaps these two might be helpful to be enabled if the issue does need to be investigated deeper:

• "echo_pool"
  

• "pool_pre_ping"
  

[MCatherine1994]

By checking the log, it returns a "ROLLBACK", wondering if the db session got some exception, will add a log here. Also some reading https://stackoverflow.com/questions/64763770/why-we-use-yield-to-get-sessionlocal-in-fastapi-with-sqlalchemy and an example to create a db session using FastAPI and SqlAlchemy tiangolo/fastapi#6006

[MCatherine1994]

Sqlalchmey has a connection pool mechanism by default. So in the get_db() method, we'll get a session for each single request. Using "yeild" so the db.close() will be called after finish all queries. When call db.close(), it doesn't mean the session is completely removed. It actually gives the connection back to the connection pool of Engine and doesn't close the connection, the session can be used again. The default connection pool size is 5. It's also possible that in the busy time, we need more than 5 connections simultaneously.

Reference:

https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session.close

https://stackoverflow.com/questions/21738944/how-to-close-a-sqlalchemy-session

[MCatherine1994]

Set up the AWS Prod CloudWatch for the DatabaseConnections for RDS proxy fam-cluster-fam-api-proxy-api and database cluster, but need more understanding of how to read the graph, and this might be not related to the sqlAlchemy connection pool size. The maximum number of connections allowed to an Aurora DB instance is determined by the max_connections parameter. However if we use RDS proxy, it should auto handle the connection pool.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.howitworks.html
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Performance.html#Aurora.Managing.MaxConnections
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Managing.html#AuroraPostgreSQL.Managing.MaxConnections

CloudWatch Dashboard in PROD for data in the past 3 month:

CloudWatch Dashboard in TEST for data in the past 3 month:

Will keep trying to see if can get this 403 error again in the future.