Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Swap JDBC for Arrow ADBC #1756

Open
jaidisido opened this issue Nov 9, 2022 · 2 comments · May be fixed by #2786
Open

Swap JDBC for Arrow ADBC #1756

jaidisido opened this issue Nov 9, 2022 · 2 comments · May be fixed by #2786
Labels
feature
Projects
AWS SDK for pandas roadmap

Comments

@jaidisido
Copy link
Contributor

jaidisido commented Nov 9, 2022 •
edited

Leveraging the Arrow ADBC Driver has a number of advantages:

  • A single, universal driver would replace all existing, proprietary drivers (mssql, postgres...)
  • Better performance due to not processing each row separately and tapping into pyarrow to avoid serialization/deserialization

Additional context
https://arrow.apache.org/docs/dev/format/ADBC.html

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.
@jaidisido jaidisido added this to the 3.0.0 milestone Nov 9, 2022
@jaidisido jaidisido changed the title Replace existing database connections with JDBC Driver for Arrow Flight SQL Swap JDBC for Arrow ADBC Jan 26, 2023
@jaidisido jaidisido moved this from To Do to Backlog in AWS SDK for pandas roadmap Feb 7, 2023
@jaidisido jaidisido modified the milestones: 3.0.0, 3.1.0 May 3, 2023
@LeonLuttenberger LeonLuttenberger modified the milestones: 3.1.0, 3.2.0 May 5, 2023
@kukushking kukushking removed this from the 3.2.0 milestone Jun 6, 2023
@jaidisido jaidisido added this to the 3.3.0 milestone Jun 6, 2023
@jaidisido jaidisido modified the milestones: 3.3.0, 3.4.0 Aug 1, 2023
@LeonLuttenberger LeonLuttenberger modified the milestones: 3.4.0, 3.5.0 Sep 20, 2023
@LeonLuttenberger LeonLuttenberger removed this from the 3.5.0 milestone Feb 20, 2024
@LeonLuttenberger
Copy link
Contributor

The implementation for this is in progress for Postgresql (see development branch). However, we are currently blocked by this bug in Pandas: pandas-dev/pandas#57539

@LeonLuttenberger LeonLuttenberger linked a pull request Apr 22, 2024 that will close this issue
Draft
@LeonLuttenberger LeonLuttenberger linked a pull request Apr 22, 2024 that will close this issue
feat: add ArrowDBC module #2786
Draft
@github-actions GitHub Actions
Copy link

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature
Projects
AWS SDK for pandas roadmap
Backlog
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: add ArrowDBC module aws/aws-sdk-pandas
3 participants
@kukushking @LeonLuttenberger @jaidisido