Provide Releng / Conda packaging for the client extensions

[bergmanngabor]

No description provided.

[bergmanngabor]

For purposes of variability / PLE, we do not provide a definitive version of the core generated Python client, and it is only distributed via the IQS instance itself. This way we won't be able to build the Jupyter extensions package (which depends on the core generated client) on public conda-forge servers. Of course it might be possible to feed it a fake version of the dependency, but then we will have to instruct our users not to touch this fake client package.

I am at a loss regarding what to do. I think the issue needs discussion.

[istvanrath]

For purposes of variability / PLE, we do not provide a definitive version of the core generated Python client, and it is only distributed via the IQS instance itself.

Why? Since we still do releases of IQS, why not do managed releases the python API package as well?

[bergmanngabor]

Why?

There are IQS instances with and without TWC support in their public API. Orthogonally, we have IQS instances with different authentication mechanisms (currently none or basic auth). Consequently, each combination requires a different "product line" of the generated Python client.

Note that in case of TWC inclusion/exclusion, it is possible to go with a single unified "superset" client, where some functions may or may not work depending on which IQS instance you connect to; though for the public demo we explicitly opted to hide unsupported features. In case of authentication, however, AFAIK it is not possible for a single version of the Python client to support both kinds of servers.

why not do managed releases the python API package as well?

The main problem is variability, not versioning.

[bergmanngabor]

Provide a conda package with basic auth, full TWC+MMS API.

[bergmanngabor]

Provide a conda package with basic auth, full TWC+MMS API.

I have updated the conda releng files and built the packages locally. Shall I send / put the built packages somewhere?

I have also started thinking about possible automation.

1. Improve automated client code generation (see existing Gradle task generatePythonClient)
   • Make sure code from the two API yamls are generated into separate folders
   • Make the code output its own thing, instead of directly emitting it into a docker project dir
2. Add a Gradle task (or some other form of CI build step) that does something like python .\setup.py sdist --formats=gztar to package each of the the two generated clients into sdist/bdist files. These may or may not be bundled with web server app, similarly to https://openmbee.incquery.io/client/
3. On release, deploy the built client sdist/bdist files somewhere, e.g. static.incquerylabs.com; one attractive option would be to twine it up onto PyPI so that they become easily available and installable via pip install.
4. Once we have "release" Python client packages available from a public URL, the wrapping Conda recipe files (this one and this one) can be updated to refer to these public URLs under the source tag.
5. At this point, building the conda packages would no longer require a development environment with the generated client code, so it could happen easily in CI, or even on conda-forge

[bergmanngabor]

@abelhegedus how shall I proceed?

[abelhegedus]

Yes, the completed artifacts should be made available. If possible, this should be a central repository (pypi and condaforge), not IQL infrastructure.

Other than that, we should create issues for follow-up steps,

• Python client generation: I think it would be cleaner if the config to generate from a target yaml would be located here
• separate clients: we can get rid of that and merge the two apis

[bergmanngabor]

Factored out follow-up issues.

Assigned to me and scheduled for next milestone:

• https://github.com/IncQueryLabs/incquery-server/issues/2562
• Document code generation for the Python client #27
• https://github.com/IncQueryLabs/incquery-server/issues/2564

Not yet assigned or scheduled:

• Automate PyPi package release #28
• Publish Conda packages #29

Closing.