VDK connectors plugin capabilities

[ivakoleva]

Hello, folks,

I want to gather ideas or proposals for capabilities of a potential vdk-connectors plugin, dedicated to
help users to discover the VDK database connectors currently available.

I am thinking of configurable {"repo1": "dir", "repo2": "dir"}, set by default to
{"https://github.com/vmware/versatile-data-kit": "projects/vdk-plugins"}

Then, (1) the plugin will dynamically fetch and scan recursively the directory, common to:
git ls-files --directory projects/vdk-plugins

Next step (2) is to parse the files and look for:

1. ManagedConnectionBase implementation
2. context.connections.add_open_connection_factory_method() connection registration that wires the implementation
3. dbtype param of context.connections.add_open_connection_factory_method() for the name of the database registered

For example:

1. class ImpalaConnection(ManagedConnectionBase) in vdk-impala/.../impala_connection.py
2. & 3. context.connections.add_open_connection_factory_method("IMPALA", ...) in vdk-impala/.../impala_plugin.py

That would be sufficient to identify if a plugin registers in vdk-core a new database connection with a particular name.

Next step (3) is to parse the directories of connectors identified, looking for @click.command provided,
to output the user each available connector's VDK CLI commands.
For example in vdk-impala/.../impala_plugin.py:

@click.command(name="impala-query", help="executes SQL query against Impala")

For more information, check out the VEP-1418 specification draft #1419

[antoniivanov]

This is specific idea only for database plugins ? Or is it for any plugin .

I do think we need better documentation of what plugions are available and what each plugin do and possibly some categorization.
Some idea is described here.

But possibly having a page like https://docs.pytest.org/en/latest/reference/plugin_list.html would be a good first step.

[ivakoleva]

I though about a general purpose plugin scraper ofc, however such is easily generalised after we implement vdk-connectors (extract commons).
vdk-connectors focuses on connector-specific features, for example vdk connect :) The purpose is to explore those common to connectors features we may want to have, what do we imagine a VDK connector could do?

[ivakoleva]

Better documentation is always helpful for observability. What vdk commands allow you to do with a connector, is a functional change.

[antoniivanov]

For database plugins btw, I've been thinking that we probably can make it easier to create a new db plugin and remove duplication across different plugins - #1444

If we have vdk-databese "base" plugin , then we can include some db connector discoverability logic there.

[ivakoleva]

Maybe we may add a db connector marker, to easier identify. However, how do you enforce it?
We may easily simply scan when connection is implemented, to determine if a connector...

[ivakoleva]

I like the idea of extracting connectors commons to a base connector plugin, lets draft one and measure if there is enough commons to be extracted