-
Notifications
You must be signed in to change notification settings - Fork 46
Contributor meetings
Manfred Moser edited this page May 30, 2024
·
46 revisions
The Trino Gateway project organizes regular meetings with contributors to improve collaboration with maintainers and among contributors. This space acts is used to plan upcoming meetings and provide meeting minutes after the event.
- Virtual meetings are held every second Wednesday, 6am PST.
- Anyone can attend.
- Contact Manfred Moser on Trino slack for an invite.
- Meeting details are also announced on slack in the trino-gateway-dev channel
- Topics can be suggested prior to the event by updating this page, message on the slack channel, or during the meeting
6am PST
- Review and discussion of slide deck for TCC
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya, oneonestar, rdsarvar
Topics, notes, action items, decisions
- Release 10 planning
- airlift refactor ideally hit before
- Slide deck for Trino Contributor Congregation
- Manfred created deck, team will edit together and review in next sync
- ping Manfred for access
- Discussed the issue around HTTPS and separate port https://github.com/trinodb/trino-gateway/pull/263
- mentioned by Will
- decided to close in favour of using only one HTTP port in airlift refactor and hence close this PR.
- Manfred has done that now
- Basically will be fixed in release 10
- Discussed how we have context paths for Trino and Trino Gateway that overlap
- Explained paths and extrawhitelist paths configuration
- Decided to move to specific context for Trino Gateway in the longer run
- For example /tgw/ui and /tgw/api and so on
- And then assume all others have to be passed through
- But that is a larger refactor for later (after airlift refactor and such)
- Vishal reports that release 9 is working well in production
- Decided to file issues for UI problems
- so we can track and work on the
- and also ping Peng in these issues
- Andy and Jaeho to work on pending state PR for review and merge
- Vishal and Will to work on Process Request PR
- manfred to add discussion topic for TCC
- unauthenticated Trino rest endpoint with number of workers and active queries - done
- Star will work on airlift removal soon
- requesting merge hold
- hopefully ready for review next week
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya, oneonestar, rdsarvar
Topics, notes, action items, decisions
- Trino Gateway 8 and 9 releases, plan towards 10
- Documentation improvements
- Discussed #222 Trino gateway health state design
- Decided to proceed with separate in-memory status per instance for now, andythsu will work on PR
- Second step to add to UI
- Also add JMX mbean
- Openmetrics to follow when airlift refactor allows openmetrics easily, will need config like in Trino
- Debate about terms "Backends" vs "Trino clusters"
- Decided as team to refactor over time to use the term "Trino clusters" instead of "Backends"
- We will implement this in the codebase and the docs
- Query details extraction PR, willmostly
- Trino 445 and airlift upgrade caused issues due to transient jetty upgrade, will back out and do that separately later
- General advice from mosabua - keep PRs smaller to enable easier reviews and prevent scope creep
- willmostly asked about helm chart PR - mosabua already reviewed and approved
- Docs update will proceed with focus on installation and cleanup, mosabua to help
- vishalya working towards getting 9 release into prod
- vishalya reviewing sql parsing PR from willmostly
- vishalya mentioned small UI bugs
- we need to file issues and then also learn more react and familiarize ourselves with codebase
- andythsu and mosabua to look a bit
- mosabua to also pull in Peng (orig author of UI)
- Trino Contributor Congregration - team to collaborate on overview and roadmap deck
- Manfred to start deck
- oneonestar found some oauth/oidc security issue and created fix PRs, mosabua to review and work towards merge before 10 release, further testing can be done by others after merge easier
- oneonestar mentioned issue that currently two ports are used for UI and API separately, airlift does not support that
- team agreed that collapsing it all into one port is fine
- reason is most likely historical to separate security concerns, but we now have authentication so no problem
Cancelled due to availability issues
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, dprophet, vishalya
Topics, notes, action items, decisions
- Discussion around Trino Gateway 8 release
- Decided to release asap even if helm chart and other changes don't make it in
- mosabua to update release notes and get things ready to work with martint
- Airlift as base
- work ongoing
- required to get plugin system off the ground
- also required for OpenTelemetry integration
- Discussion around smarter routing based on query and security info
- definitely something we are working towards,
- first effort in https://github.com/trinodb/trino-gateway/pull/325
- more to come, also including info of queried catalogs, schemas, and more
- in future potentially around query plan data including cost and query shape
- also discussed trino2trino connector and concluded that the two separate efforts are related and complimentary, so we will pursue both
- Cluster status
- Related to https://github.com/trinodb/trino-gateway/pull/163
- Need to support multiple Trino Gateway instances
- Store status in db and cache on nodes
- willmorrison walked through example guava cache code
- andythsu will create issue and close existing PR
- mosabua to potentially replace https://github.com/trinodb/trino-gateway/pull/263
- Docs site still in progress with mosabua
- we are looking to get more details for deployment advice
6am PST
Attendees
mosabua, Chaho12, andythsu, willmostly, vishalya,
Topics, notes, action items, decisions
- Trino Gateway 7 release done, working towards 8 soon
- HTTPS issue, agreed to merge before 8
- Client host redirect PR in Trino - oneonestar mentioned that this is not yet ready, mosabua mentioned it might therefore not make it for Trino 444, @electrum will review
- Agreed to merge router module PR, and get separate PR to follow up on code style, done by mosabua
- JDBI refactor - reviewed now, agreed to merge after router module
- mosabua to drive HTTPS PR to merge https://github.com/trinodb/trino-gateway/pull/263
- mosabua to replace https://github.com/trinodb/trino-gateway/pull/217/ with Trino config tips section, done in https://github.com/trinodb/trino-gateway/pull/297
- andythsu will get release 7 in prod soon and share info
- Chaho12 updated quickstart PR, mosabua to finalize https://github.com/trinodb/trino-gateway/pull/247, done
6am PST
Attendees
mosabua, vishalya, Chaho12, andythsu, oneonestar, wendigo, nineinchnick
Topics, notes, action items, decisions
- Trino Gateway 7 release about to be cut, release notes and codebase are ready
- Agreed to upgrade to Java 21 after 7 release - https://github.com/trinodb/trino-gateway/pull/225
- Doc content and site use has no progress, but work will continue soon https://github.com/trinodb/trino-gateway/pull/254
- Email notification PR needs update and review - https://github.com/trinodb/trino-gateway/pull/244
- Once new release is available, andythsu will provide feedback about production usage
- Agreed to get router module PR merged right after 7 is cut - https://github.com/trinodb/trino-gateway/pull/98
- Agreed to get ActiveJDBC removal/JDBI refactor merged right after 7 is cut - https://github.com/trinodb/trino-gateway/pull/284 and https://github.com/trinodb/trino-gateway/pull/287
- mosabua to set up topics in docs for production deployment requirements and tips
- vishalya to start rate limiting high level discussion
- mosabua to follow up about https://github.com/trinodb/trino/issues/21026 at contributor call and in general
- mosabua to plan for faster follow up Trino Gateway 8 release and higher cadence going forward
6am PST
Attendees
mosabua, willmostly, vishalya, Chaho12, andythsu, oneonestar, [alaturqua])(https://github.com/alaturqua)
Topics, notes, action items, decisions
- Discussed UI PR, decided to potentially merge after final review, probably ready to merge
- Discussed v1/info PR, mosabua to review and potentially merge, also discussed potential changes in Trino itself for that end point and wish for more info from a Trino endpoint such as GC pauses, memory and CPU pressure, nubmer of workers and more, mosabua to talk to others
- Query Id extraction PR is ready, willmostly will add javadoc and merge
- Cookie based routing needs reviews, mosabua will look
- Configurable routing modules - vishalya will update PR to get ready
- mosabua to start doc to assemble ideas for cluster health data
- oneonestar lead discussion around support for certificate auth, mosabua and others to chime in on issue
- future feature "query retry" needs design doc or ticket and more, needs to be initiated
- sql rewriting comes up as feature request, potentially good to expose SPI for others to add whatever rewriting they need, mosabua to start issue for gathering use cases and more
- Topic of preprocessing of query for better routing rules was discussed, definitely interesting future feature, potentially share parse tree
- Topic of result caching with redis and/or object storage as cache was discussed, mosabua and willmostly to continue discussion with other
6am PST
Attendees
mosabua, willmostly, dprophet, vishalya, Chaho12, sbernauer, rdsarvar
Notes, action items, decisions
- Trino Gateway 6 is out, including first docker container
- Congrats to our new subproject maintainers willmostly, vishalya, and Chaho12
- Report from PoC with MkDocs
- Chat about about Trino healthcheck REST endpoind, mosabua to follow up with Trino maintainers
- Decided to put focus on pluggable policies and the PR from vishalya before we implement further routing policies
- New UI testing went well as reported by Chaho12 and vishalya, decided to work with contributor to get to merge, authentication will have to become optional
- Discussed impact on Oracle support (and in general) of the refactor from ActiveJDBC to JDBI, will continue refactor and figure out specific solution
- State management for multipole instances - to be determined but probably via database, maybe add Redis as supported backend, no current task
- Discussion about automatic retry of queries in the future, will need storage of submitted queries and state, and more, will work on that at a later stage
- Trino LB demo and lots of good discussion with sbernauer, for example scripted polices via some JVM scripting support
6am PST
Proposed topics
- Documentation setup (MkDocs, Docusaurus, ..?)
- PR reviews and status
- Docker setup and container image
- Next release
Attendees
mosabua, willmostly, dprophet, andythsu, vishalya, Chaho12, sbernauer, soenkeliebau, rdsarvar
Notes, action items, decisions
- Round of introductions from everyone including current status/tasks
- Discussion of current ongoing PRs, various reviews and rebases are in progress, we will try to get as many done as possible before we cut new release
- Java 21 discussion - decided to cut at least version 6 and 7 before we go to Java 21, no current benefit to move
- New UI - decided to ensure testing of the new UI is done by a number of contributors before we merge
- Trino Gateway Healthcheck - filed issue, we should add Dropwizard standard healthcheck and then use it for container start
- Trino healthcheck and cluster status - discussed approach since a healthy cluster must have at least one worker and the current REST endpoints dont show that info, mosabua to talk with Trino maintainers about potentially adding another public endpoint or expand /v1/status or /v1/info with worker count or so, for now we will potentially have to use UI endpoint (and need to figure out authentication with that) or use JDBC or something else
- Plan for new release after some more merges, definitely before next sync
6am PST
Proposed topics
- Recent code changes towards version 5 release
- Onboarding new maintainers
- New UI
- Docker container work
- Java 17 or migrating to Java 21
- Trino cluster healthcheck design
- Improve Trino JDBC driver usage
- Dependency cleanup
Attendees
mosabua, willmostly, dprophet, andythsu, vishalya, Chaho12, nil1729, sajjoseph
Notes, action items, decisions
- Round of introductions from everyone including current status/tasks
- Discussed plans towards next releases
- Reached agreement to merge OAuth2 PR and then cut release 5
- Docker, new UI, Oracle and others all after the release
- Onboarding for subproject maintainers in progress
- Agreed to move to Java 21 soon, need to verify dependencies and functionality, and update deps
- Discussed plans for docs to move to sphinx and github pages over time
- Agreed to look at new UI and move towards it after review and some testing
- Agreed to work towards pluggable router policies
- Did not find time to talk about cluster healthcheck design and distributed healthchecks
- Agreed to use github issues for roadmapping and discussion about design for new features
- Agreed to have separate meetings and discussion for specific topics as needed
- Agreed to use trino-gateway-dev channel for communication
- Agreed that meeting time and schedule every two weeks is good
- Project will work towards smarter routing with explain plan/cost, client tags and other approaches, specific cluster only routing for oauth
- Discussed need for metadata caching to tackle traffic very noisy BI tools, considering this to also be in scope of the project
- Discussed deployment modes with cold-standby vs multiple stateless nodes and will work on details more
Prior events include various calls and meeting in the creation and start of the project.