libghdl documentation

[m-kru]

@tgingold I was analyzing ghdl_ls and it looks like there is no documentation for libghdl. Few questions came into my mind. Is libghdl thread safe? Can multiple programs interact with libghdl simultaneously? Is it safe to use ghdl_ls when there is some simulation running in the background? As libghdl is a shared library it should be possible to document its API. It would be really nice if the documentation was placed in the source files, and later fetched to the online docs. My personal feeling is that libghdl makes it possible to use the power of GHDL in Python, however it is not easy to start, as without documentation user still does not know what libghdl exactly is and how it works. I could even try to prepare some libghdl documentation, but I do not know where to start, what are the source files I should be interested in.

One more question. Just out of curiosity. Most functions from libghdl have double floor in names, example libghdl.libghdl__set_hooks_for_analysis(). What does imply this double floor between libghdl and set?

[eine]

FTR, the docs of GHDL are generated with Sphinx, which was primarily designed and is mostly used for documenting Python projects. Hence, we can generate "nice" docs from docstrings written in the Python sources, and have them integrated in the docs cleanly (much better than how we document GHDL itself). See, for example, the documentation of VUnit's Python interface: http://vunit.github.io/py/ui.html

Why don't we do it for Ada and VHDL sources too? Because there is no built-in mechanism in Sphinx to do so. See #484 and https://github.com/Paebbels/sphinxcontrib-vhdldomain.

[m-kru]

To be honest, I am not sure what should be documented, functions from shared library in Ada files or maybe Python bindings. It looks like right now Python bindings do not add any functionality or abstraction, example libraries.py:

from libghdl import libghdl
from ctypes import c_int32

Get_Libraries_Chain = libghdl.libraries__get_libraries_chain

Add_Design_Unit_Into_Library = \
    libghdl.libraries__add_design_unit_into_library

# Use .value
Library_Location = c_int32.in_dll(libghdl, "libraries__library_location")

# Use .value
Work_Library = c_int32.in_dll(libghdl, "libraries__work_library")

Purge_Design_File = libghdl.libraries__purge_design_file

Find_Entity_For_Component = libghdl.libraries__find_entity_for_component

Get_Library_No_Create = libghdl.libraries__get_library_no_create

Find_Primary_Unit = libghdl.libraries__find_primary_unit

By the way, these names look a bit unPythonic.

I also do not know if Python bindings should provide some extra functionality or should they only directly call functions from shared library.

[m-kru]

I made some preliminary analysis of possible solutions. In my opinion all functions, variables etc. available in the shared library should be documented in the Ada sources. Any bindings to libghdl should be raw bindings, no additional functionalities, only direct mapping onto the original API. In such case we can have single documentation for bindings for different languages, plus it is easier to maintain bindings compatibility.

[eine]

In my opinion all functions, variables etc. available in the shared library should be documented in the Ada sources.

Yes, that would be ideal. However, on the one hand, we don't have resources to do it. Otherwise, it would already be done. The single person that can currently do it from scratch is Tristan, and I think we all agree that he is ok prioritizing other missing/incomplete features (synthesis, VHDL 2008). On the other hand, even if sources were documented in Ada, a priori we could not create an HTML/PDF documentation from there (see #484). This might have changed with latest versions of GPS. Should generating docs à la Doxygen with GPS be possible, I would vote for that.

For now, the most viable solution is to write the docs in rst from scratch (as it is done for every GHDL command and option). Alternatively, docstrings can be written in Python sources, which would be automatically picked by Sphinx, so we would avoid writing the hierarchy/structure of modules manually.

So, in this context, I think a good approach would be to document the functions that are needed/used in vhdl_langserver only.

Any bindings to libghdl should be raw bindings, no additional functionalities, only direct mapping onto the original API.

That sounds good. In this case, it makes more sense to use docstrings, given the context explained above. That is, the docstrings in libghdl would be the documentation of the API, and comments in Ada sources would be considered internal docs. So, the docstring explains what it does and how to use it, and provides a ref to the Ada source where the users/developers can see how is it actually done/implemented.

What about your concern regarding names being unpythonic? Do you think it is acceptable to keep them exactly as in Ada? Or is it reasonable to do slight style changes for them to feel more natural in the environment? For example, in PyGitHub, which are bindings for GitHub's API, names are slightly different.

[m-kru]

On the other hand, even if sources were documented in Ada, a priori we could not create an HTML/PDF documentation from there

I do not think this is a problem. Regular user is not interested in shared library documentation. The online docs can redirect to the Ada sources for documentation.

For now, the most viable solution is to write the docs in rst from scratch (as it is done for every GHDL command and option).

I do not have good experience with keeping sources and docs in separate files. It is easy to forget to update the docs.

Alternatively, docstrings can be written in Python sources, which would be automatically picked by Sphinx, so we would avoid writing the hierarchy/structure of modules manually.

What if someone tries to write bindings for different language, for example Rust? Then you would have to maintain more than 1 docs.

What about your concern regarding names being unpythonic? Do you think it is acceptable to keep them exactly as in Ada? Or is it reasonable to do slight style changes for them to feel more natural in the environment? For example, in PyGitHub, which are bindings for GitHub's API, names are slightly different.

To bo honest I do not know. I just pointed out they are unPythonic :).

[eine]

I do not think this is a problem. Regular user is not interested in shared library documentation. The online docs can redirect to the Ada sources for documentation.

I think we are mixing terms... One thing is the documentation, meaning the HTML and the manual in PDF, which are both generated from the same rst sources. A different issue is to have a commented codebase. Commenting all the Ada sources is, as explained, unaffordable ATM. OTOH, keeping dozens of references between the online docs and the sources is the same effort as writing the docs in separate files; i.e. something needs to be kept in sync, regardless of where are the 5-10 lines of explanations written.

I do not have good experience with keeping sources and docs in separate files. It is easy to forget to update the docs.

That's why #484 exists, and why I would prefer to use Doxygen, GPS or any other ready-to-use mechanism that allows to extract comments and cross-references from the sources. Unfortunately, that is not possible ATM. Comments can be written, but there is no possibility for formatting the params/args, or cross-referencing functions/variables.

So, as soon as you need some comments to reference others, you need to move those refs to rst/sphinx. That's why I said above that the effort is similar.

What if someone tries to write bindings for different language, for example Rust? Then you would have to maintain more than 1 docs.

The documentation is unique, as it matches 1-to-1 the API available in the shared lib. Then, for each function, you can have a list that indicates the name of the binding in each language. More than 1 docs are only required if the bindings of some language do not match 1-to-1. And even in that case, it can be explained as a comment.

Overall, I'm not proposing to write the docs in Python because the bindings are in Python. I am proposing it because the documentation system is Sphinx. If we were using Jekyll, Hugo or AsciiDoc, my proposal would probably be different.

OTOH, the default documentation generators of some languages are good but feature limited. For example, golang can generate nice looking doc sites automaticallly. However, it is not possible to export those sites, so it's impossible to have godocs integrated in any other site. It would need to be uploaded as a separated sub-site, and then linked from the main docs. I am not familiar with Rust's built-in docs generation. Should it allow integrating in other sites, it would make sense to document those bindings as an alternative to writing the docs in Python. Otherwise, I would leave the bindings undocumented/uncommented and refer to the Ada/Python docs.

If all you want is some comments to be added in some of the Ada sources, I suggest to take #1249 as a reference. That is, try to add the comments yourself and then discuss the ones you don't understand. If your proposal is more related to the high-level architecture and usage of the shared library, I believe it'd fit in https://ghdl.readthedocs.io/en/latest/internals/Overview.html.

[tgingold]

The current python binding is just a python interface of the existing Ada code. That explains why the names are unpythonic (the names used are the same as the names in the Ada core) and why there is no documentation (no duplication).

For the doc, you need to refer to the Ada sources. Just go to the function/variable with the same name in the source file with the same name (except the extension).

I think the next step is to create a pythonic interface based on the thin interface. It requires a bit of design, but there is a good news: nodes.py, which is the biggest file, the one that describes the AST is automatically generated. So part of the pythonic interface could also be automatically generated.

[umarcor]

While working on #1449 I found that python/libghdl/thin seems to depend on python/libghdl. So, the dependency (libghdl) is the parent of the dependent (thin). This feels weird. I would expect thin to be either a sibling or a parent of libghdl.

Precisely, I was trying to simplify the following imports in show_units.py and show_ports.py:

import libghdl
from libghdl.thin import name_table
from libghdl.thin import files_map
from libghdl.thin.vhdl import nodes
from libghdl.thin.vhdl import sem_lib

to either

from libghdl import name_table, files_map, nodes, sem_lib
# or
from thin import name_table, files_map, nodes, sem_lib
# or
from libghdl.thin import name_table, files_map, nodes, sem_lib

However, adding the following to python/libghdl/__init__.py is not possible (because thin depends on libghdl):

from libghdl.thin import name_table
from libghdl.thin import files_map
from libghdl.thin.vhdl import nodes
from libghdl.thin.vhdl import sem_lib

ImportError: cannot import name 'libghdl' from partially initialized module 'libghdl' (most likely due to a circular import)

Alternatively, I tried adding the following to python/libghdl/thin/__init__.py:

from thin.vhdl import nodes
from thin.vhdl import sem_lib

or

from vhdl import nodes
from vhdl import sem_lib

None of them work:

ModuleNotFoundError: No module named 'thin'

ModuleNotFoundError: No module named 'vhdl'

The only solution I found is:

from libghdl.thin.vhdl import nodes
from libghdl.thin.vhdl import sem_lib

As a result, users can reduce their import block to:

import libghdl
from libghdl.thin import name_table, files_map, nodes, sem_lib

[umarcor]

FTR, the organisation of Python sources was reworked in #1556. See also #1557.

[Paebbels]

@m-kru I'm adding now documentation to pyGHDL.libghdl.

I'm sorry, currently the Navigation bar is broken when browsing inside of auto generated documentation pages, but when you stay in the main window, you can also ready browse a lot.

http://paebbels.github.io/ghdl/pyGHDL/pyGHDL.libghdl.html

The documentation will be:

• taken from Ada source for manually written adapter code
• enhanced and adapted to Python needs
• using ReST with cross references, etc.
• add type annotations

See this example: http://paebbels.github.io/ghdl/pyGHDL/pyGHDL.libghdl.files_map_editor.html

Currently the Sphinx plugins seem to have problems with user-defined type variables like SourceFileEntry.

---

When working on that topic, I tried to improve the libghdl interface to ease handling of libghdl. Anyhow, pyGHDL.libghdl is the low-level part doing the binding to the shared library. It offers also some abstraction like str to char* conversions and generators. Everything else is planned to be in either pyGHDL.dom or pyGHDL.lsp, ...

---

For your questions:

One more question. Just out of curiosity. Most functions from libghdl have double floor in names, example libghdl.libghdl__set_hooks_for_analysis(). What does imply this double floor between libghdl and set?

_ _ is used to preserve the hierarchy from Ada sources. E.g. vhdl__scanner__scan() refers to vhdl/scanner.ads::scan().

[m-kru]

@eine @Paebbels

I am starting to feel lost.
There is (is going to be) libghdl with Python bindings, pyVHDLParser and pyVHDLModel.
Is it described anywhere which one should be used depending on what user tries to achieve?
Is pyVHDLParser going to utilize libghdl? As far as I remember it has its own parser.
Especially the border between libghdl and pyVHDLParser looks blurred.

[Xiretza]

As far as I understand, pyVHDLParser and libghdl don't have anything to do with each other. pyVHDLModel is just a set of data structures that can be implemented by VHDL-processing python libraries, such as pyVHDLParser or libghdl.

[eine]

@m-kru, the layer between pyGHDL.libghdl and pyVHDLModel is pyGHDL.dom. This one imports pyVHDLmodel and generates "objects" by extracting the data from pyGHDL.libghdl. For final users, pyGHDL.dom is "pyVHDLModel with pyGHDL.libghdl aka GHDL backend".

pyVHDLParser is unrelated to GHDL or pyGHDL.libghdl. That is "another backend". So, if a user wants to obtain the pyVHDLModel of their VHDL sources, the have (will have) two options:

• Use pyVHDLParser as a backend, which is a Python only solution but less mature.
• Use GHDL as a backend, which is a compiled tool (needs to be installed), but is more mature and used.

Other parsers, such as rust_hdl, @Nic30's ANTLR based infrastructure (which has Python bindings), etc. can also provide a pyVHDLModel compatible module. Therefore, the purpose of pyVHDLModel is to avoid the fragmentation in higher level features: linting, syntax, LSP, style, sphinx/asciidoc documentation generation... All those projects should not need to decide which parser to use, any should work.

pyVHDLModel was previously part of pyVHDLParser. In December, it was split to vhdl/pyVHDLParser and GHDL's python modules were overhauled accordingly. I believe that is what might be misleading for you.

[umarcor]

As commented in #1449, pyGHDL.dom is now used in the following demos:

• Open Source VHDL Design Explorer (OSVDE): https://umarcor.github.io/osvb/apis/project/OSVDE.html
• pyVHDLModelUtils: https://umarcor.github.io/osvb/apis/project/pyVHDLModelUtils.html
• Documentation generation: https://umarcor.github.io/osvb/apis/project/DocGen.html

We hope that explicit code examples such as https://github.com/umarcor/osvb/blob/main/mods/pyVHDLModelUtils/sphinx.py#L56-L79 can be useful for others to tinker with pyVHDLModel.