Failed worker doesn't handle requests properly on recovery #21735
Comments
|
HTTP/2 is not really tested or supported (at least for now). We've been reporting similar issues to the jetty team and they are steadily being fixed. NPE usually means "use after free" which suggests that request/response was already recycled but some parts of the code still try to use it and access/write. This could happen if the jetty client i.e. aborts the request due to a timeout. There are bunch of fixes for this scenarios that will be a part of the next jetty release that we will update to shortly after. |
|
Thanks! Are you considering Jetty 12.0.9 or higher? Jetty 12.0.9 is out and airlift 247-SNAPSHOT is still configured to use 12.0.8.. I am using 12.0.8 and still seeing the above issue. Are we planning to upgrade to 12.0.9 or its next release? |
|
@sajjoseph next release |
|
@sajjoseph actually Jetty 12.0.9 is not yet out. It's tagged but artifacts are not yet published |
|
cc @losipiuk |
|
@wendigo - I tried 12.0.9 snapshot version and looks like jetty problems are fixed now. I don't see the above exceptions anymore. But, I still see that the restarted worker node is not doing any work as the assigned task to that worker is staying in "PLANNED" state. All other workers finished their work. Eventually, the query fails with the following PAGE_TRANSPORT_TIMEOUT exception: @findepi / @losipiuk / @wendigo Looking forward to your assistance. |
|
@sajjoseph are we taking about FTE or pipeline execution? |
|
We are not using FTE. Thanks! |
|
Quick update. I will be upgrading the cluster to the latest (447) tonight. I am not anticipating any issues though. |
|
@sajjoseph glad to hear positive feedback on the #21744 PR!
I am interesting in this scenario. I was not able to reproduce this particular problem on current master (before #21744) and this problem wasn't the motivation for that PR. Now that master is updated to jetty 12.0.9 I will try to reproduce #21735 (comment) again with a different test query. |
wasn't able to repro with a query involving ORDER BY (from the issue description) with and without HTTP/2 for internal comms. |
|
I used a heavily modified version of query replay tool to re-run production traffic in a separate cluster. I purposefully kept just one worker in that cluster so that I can make the worker node fail with load (I was able to do that consistently with 444 version). After restart, the worker node was getting into the state I described above. After #21744, the worker is not failing even after seeing 100s of concurrent queries (many queries were failing due to memory pressure as the worker can't handle the data volume, but the worker itself didn't go down this time!!! Kudos to Trino 447, JDK 22, Jetty 12.0.9, Airlift 248, HTTP/2 etc). I am testing it with even more queries and see if I can make it fail. Even though worker didn't fail, I manually restarted it multiple times (in the middle of validations) and it just joined the cluster and started processing fresh queries that came in. This is why I feel that #21744 is helping with the above issue. I will keep you posted. |
|
Potentially related #18329. |
Environment: AWS EC2 using Graviton
Version: Trino 444
JDK: jdk-21.0.1+12
Jetty: 12.0.8
HTTP mode: HTTP/2
One of the worker processes failed (due to memory pressure I believe) in our environment and the Trino process on that worker was restarted. Everything was normal and there were no issues detected during restart. Once it joined the cluster, we ran the following query.
SELECT TABLE_CATALOG as TABLE_CAT, TABLE_SCHEMA AS TABLE_SCHEM, TABLE_NAME, TABLE_TYPE FROM <catalog>.information_schema.tables WHERE TABLE_CATALOG = '<catalog_name>' AND TABLE_SCHEMA LIKE '<schema_name>' ESCAPE '\' AND TABLE_NAME LIKE '<table_name>' ESCAPE '\' AND TABLE_TYPE IN ('TABLE', 'VIEW', 'BASE TABLE') ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME;This is when we noticed the following:
Even though it is a metadata query, all workers are joining this query in Stage 1 and Stage 2 is coordinator. Coordinator is returning the metadata to one of the workers and other workers in Stage 1 are finishing except the bad worker who remains on "PLANNED" state.
At the same time, the worker node had a bunch of null pointer exceptions in its server.log file.
Finally, the above query will fail with PAGE_TRANSPORT_TIMEOUT error against the bad worker.
Stack trace:
I cannot restart the coordinator to see if it will let the restarted worker node stay healthy. NPE in the log file is little suspicious. If I perform graceful shutdown on a healthy worker and restart it, it doesn't suffer from the above behavior. All workers are configured identical. I even rebooted the bad worker node to see if it will become healthy and it doesn't.
I wonder whether the coordinator or other workers retain some reference to past failed queries/tasks and requests for an update as soon as the bad worker rejoins the cluster as it retains the original ip address.
@wendigo - please see if the above scenario makes sense to you. I suspect that something is still lurking in the cluster about this worker node that any query uses it will fail.
@sopel39, @findepi - Have you seen similar issues?
The text was updated successfully, but these errors were encountered: