Proposal for Annotation Declarations (XQuery only)

[rhdunn]

This is part of work to make annotations more accessible within XQuery. The other parts are adding them to the type system and adding functions for querying them.

[michaelhkay]

I find it hard to imagine anyone actually using this. If the processor is going to make use of the annotations, then it will presumably validate them anyway. If it isn't going to make use of the annotations, then it seems to make little difference whether they are valid or not.

[rhdunn]

Some examples that immediately come to mind are:

1. https://github.com/xquery/rxq -- a RESTXQ implementation for MarkLogic; MarkLogic does not know about RESTXQ and the associated annotations;
2. portable unit test frameworks (e.g. %test, %setup, %teardown);
3. authentication -- restricting users/groups to certain REST API calls;
4. auditing (logging whenever a function is called, esp. for REST APIs).

This is expecially useful if the XQuery processor provides a way for accessing and querying annotation information like MarkLogic does. So in conjunction with a portable standard annotation API they could be useful for working with user created annotations, not just those supported by the XQuery processor.

[adamretter]

So I can see that this has some value in terms of static checking, i.e. catching errors early. But I am not sure I see the bigger picture just yet...

At the moment eXist-db's RESTXQ implementation already searches for the annotations at compile time, it will raise errors if the RESTXQ annotations are not valid. This is very much done statically, i.e. before any query is executed. So do we really need annotation declarations?

I also wonder, how the linkage would work. For a library like RESTXQ where should it define its annotations? in an XQuery code module? How are they make be available for a user working with them? Does the user need to import a module written in XQuery? Currently they are defined externally to the XQuery itself in Java in a not dissimilar way to external functions.

Finally I am not at all sure what the purpose of Named Annotation References is? Why would I want to pass a reference to an annotation?

[rhdunn]

Regarding named annotation references, I was thinking about the possibility of passing them to standardised annotation introspection functions in addition to the possibility of passing QNames to make the code more concise. For example:

if (exists(ann:annotation($f, %rest:POST#0))) then ...

[rhdunn]

Regarding the bigger picture, think about user-defined annotations instead of vendor-defined annotation. For example, MarkLogic provides vendor-specific APIs for accessing the annotations of a function declaration. It would be useful to have these statically checked in addition to the vendor-specific annotations.

[adamretter]

@rhdunn okay interesting. So at the moment we have an XQuery library module in eXist-db called "inspection" which allows you to find the functions in an XQuery module, and inspect a single function. It returns its description of a function as an XML document. That document includes details of annotations. This is all done dynamically however, as we have to compile the XQuery module before we can inspect its function signatures.

The module if you are interested is InspectionModule but its documentation is quite poor, so if you have any questions just ask.

The static idea of annotations is kinda interesting. I wonder if this doesn't fit into a bigger question of dynamic reflection vs static typing though. With dynamic reflection we can interrogate many more things such as function signatures, and declared variables and options. There are some advantages to doing things statically though, such as strict "compile time" checks; if there is a "compile time"!

[rhdunn]

I'm also thinking about enabling support in editors (such as my XQuery plugin) and static analysis tools -- having the annotations defined statically will allow me (and others) to provide better validation of annotation use.

It would also help with generating documentation (xqdoc, the BaseX wiki, the RESTXQ and other specifications), as it would allow xqdoc documentation to be attached to the annotations, and to specify the exact signature of annotations in the documentation/specification output without any confusion over the form the annotations take.

I agree about being able to access the information dynamically -- which is where an annotation/inspection/reflection function module proposal would come in (like there is for arrays and maps).

The idea is to ultimately support both static typing (this proposal) and dynamic reflection (the annotation type and an annotation function library proposal), just like is available for functions.

[duncdrum]

I like this and the related #10 proposal, and can see use-cases for making annotations first-class citizens. Together these two could lead to simpler rules for how processors should interact with annotations, more specific error types, etc.

[adamretter]

@rhdunn I am wondering how you see linking working. So if a user declares their annotations statically in a library module, can that module be imported into other modules which use those annotations?

If so, an IDE would need to parse all imported module recursively when opening any module? Perhaps that is already the case, I am not sure...

[rhdunn]

I'm thinking it would work like statically-known functions. I talk about statically-known annotations in the proposal. A library (restxq, a unit test library, etc.) would be imported into another library/main module and the annotation declarations from that library would be accessible via the statically-known annotations.

An IDE would need to parse the modules and implement logic for resolving and validating annotations, just like it would when resolving functions to their declarations and validating arguments. It's complicated ensuring the logic works both in terms of the IDE requirements, and the XPath/XQuery requirements. I have made improvements in this for the latest development version of my IntelliJ plugin which fixes some issues regarding function resolution.

The way IntelliJ works is that a language plugin implements a Lexer and Parser for the language. The parser then generates a PSI (Program Structure Interface, similar to an AST). These PSI trees can then be cached and used to implement features like variable and function lookup. The cached version may or may not use the parsed tree, for performance.

In my plugin, I have modelled the PSI tree on the EBNF grammar for XPath/XQuery. I am then using a collection of interfaces and extension method functions to implement the XPath/XQuery semantics for resolving the statically-known namespaces, expanding QNames, resolving the namespace URIs to the corresponding module prolog, and then looking up the function declarations in those prologs. This is to avoid issues with caching when editing a file in the IDE. As I said -- it's complicated :).