Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Apache Pinot® as supported database #637

Open
gAmUssA opened this issue Mar 22, 2024 · 5 comments
Open

Add Apache Pinot® as supported database #637

gAmUssA opened this issue Mar 22, 2024 · 5 comments
Assignees
Labels
databases enhancement New feature or request
Milestone

Comments

@gAmUssA
Copy link

gAmUssA commented Mar 22, 2024

Description
As part of our ongoing efforts to expand the range of databases supported by the Kotlin DataFrame project, I propose adding support for Apache Pinot®. Apache Pinot® is a real-time distributed OLAP datastore designed to deliver scalable real-time analytics with low latency. It can ingest data from batch and stream data sources in real time, making it a versatile choice for real-time analytics applications.

import org.jetbrains.kotlinx.dataframe.io.DatabaseConfiguration
import org.jetbrains.kotlinx.dataframe.api.print
import java.sql.DriverManager

val url = "jdbc:pinot://localhost:9000?brokers=localhost:8099"

val dbConfig = DatabaseConfiguration(url)

var dataFrame = DataFrame.readMySqlTable(dbConfig, "order_items_enriched", 10)
dataFrame.print()

getting error

Unsupported database type in the url: jdbc:pinot:localhost#8099. Only H2, MariaDB, MySQL, SQLite and PostgreSQL are supported!
java.lang.IllegalArgumentException: Unsupported database type in the url: jdbc:pinot:localhost#8099. Only H2, MariaDB, MySQL, SQLite and PostgreSQL are supported!

@Jolanrensen Jolanrensen added enhancement New feature or request good first issue Good for newcomers labels Mar 25, 2024
@zaleslaw
Copy link
Collaborator

@gAmUssA thanks for the suggestion, we will wait a little bit for a vote from the community regarding this issue

@gAmUssA
Copy link
Author

gAmUssA commented Mar 25, 2024

@zaleslaw could you give a background why specific implementation is required? Why generic jdbc isn't option?

@zaleslaw
Copy link
Collaborator

@gAmUssA I am not sure for Apache Pinot, but for 4 supported databases we have de-facto different implementations and known bugs for JDBC protocol implementation in type mapping. And we decided to be accurate and support and test type generation step-by-step

@gAmUssA
Copy link
Author

gAmUssA commented Mar 25, 2024

@zaleslaw it seems like there is some confusion regarding the term "vote from community". Can you please help me understand what it means? On another note, while using a library for data science and data engineering, I noticed the absence of analytical data stores such as BigQuery, ClickHouse, or Druid and obviously Pinot, I believe having access to such tools could greatly enhance the library's capabilities and make it more useful for data professionals.

@gAmUssA
Copy link
Author

gAmUssA commented Mar 25, 2024

@zaleslaw I'm open to contribute pinot support to Dataframe framework. Could you kindly point me to resources that will alow me to do it. Are there Compatibility tests available to validate database implementations?

@zaleslaw zaleslaw removed the good first issue Good for newcomers label Mar 25, 2024
@zaleslaw zaleslaw added this to the 0.14.0 milestone Apr 23, 2024
@zaleslaw zaleslaw self-assigned this Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
databases enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants