Feature request: allow additional, not-specified parameters

[j-maly]

I would like to my function to be called regardless of the parameters in the URI. If I create the following annotation:

    %rest:path("/myap/{$collection}/myfile")
    %rest:GET
function foo($collection) 

I can indeed add aditional parameters and use the uri
/myap/col/myfile?p1=1&p2=2
and it is trapped by function foo, but I don't have access to the values of the parameters. I can not use %parameter annotations, because the names of the parameters is not fixed.

So, if there is not already a way to achieve this, I propose to add another kind of annotation, which would work like this:

    %rest:path("/myap/{$collection}/myfile")
    %rest:query-param("p1", "{$p1}") 
    %rest:query-additional-params("{$additional}")
    %rest:GET
function foo($collection, $p1, $additional as map(xs:string, xs:string)) 

Here, the url
/myap/col/myfile?p1=1&p2=2
is trapped by function foo, query parameter p1 will always be passed to the parameter $p1 and query parameter p2 or any other parameter unspecified by query-param annotation is added as an entry to a map and this map is passed to the parameter $additional.

[ChristianGruen]

The Request Module [1,2] may provide a solution for that featrue request.

Hope this helps,
Christian

[1] http://exquery.github.com/expath-specs-playground/request-module-1.0-specification.html
[2] http://docs.basex.org/wiki/Request_Module

[adamretter]

Hi Jakub,

As Christian pointed out the Request Module would be the mechanism for this, however it is not implemented in eXist yet because the spec is not stable. The implementation is probably only a couple hours work, however there is one last piece for the spec that really needs to be figured out before we can call it a first draft.

Basically the spec needs to be re-worked to take account of multipart http requests.

1. We need some way of addressing the content body of each part of a multipart request.
2. Some existing functions need to be adapted to be able to access aspects of any part of the multipart request i.e. request:header-names and request:header

I am a bit short of time at the moment, so would welcome any contributions to this work, just have a chat with us and send a pull-request.

--- for later ---

In addition, and this is just an idea at the moment (and may be going to far), I think it could be interesting to discuss whether the module should in fact operate on an abstraction from the request. Rather than operating on the http request directly, we could say that it operates on a model of the http request. I am thinking of the XML model used in the EXPath HTTP Request module. In this way these functions could actually operate on that model i.e. that would be passed in as a parameter to each function. We would need an additional function to extract that model from the actual incoming http request, so that you actually work on a real incoming http server request of course, but this would just be passed into the other functions.
The only problem with this is that serializing and de-serializing an incoming HTTP request in/out of XML is incredibly expensive. However, I wonder if implementations could not get around this, by only serializing/de-serializing when absolutely nessecary,i.e. if the model generation function is not passed to another request module function.

e.g. If we imagine the functions -

request:model() as document-node(element(http:request))

request:parameter($model as function() as element(http:request), $parameter-name as xs:string) as xs:string*

Then by specification, the first function must produce an XML element containing the HTTP Request, but I wonder if an optimisation would not be for implementations to short-circuit that when used in another request module function, e.g. the second function. The spec would always ensure the thing works, but implementers could be more performant by performing an optimisation and not generating the serialization if they dont need to.

So why would we want to do this? Well it would allow you to use the request module functions on XML models that you construct for use with the EXPath HTTP Request Module. So perhaps better interoperability, although in reality, I think processing the http:response from that EXPath module is more interesting!

Perhaps there is a better way to model this in standard XQuery terms?

I wonder if Christian has any thoughts?

[msokolov]

I think the approach of supplying an xml model encompassing the http request parameters is a good one: in many ways more attractive than the functional interface, or at least, complementary to it. It leverages the strengths of an XML processing environment, and would enable, for example, transformations to be applied, in the case of a filter that might perform url rewriting or parameter aliasing. Adam - is your concern about performance related to processing the request body primarily? I wonder whether providing a model sans body (ie url, query parameters, headers and wrapper info only) would be better? The body could be available only via function call. This seems necessary anyway in order to handle multi-part, chunked encoding, and file upload, since the processing of these might be under application control right? At any rate, lazy evaluation would certainly help, whether the body is included or noe.

[adamretter]

I am not sure of the need for url rewriting or parameter aliasing, do you have a use-case that cannot be met by a Request Module and RESTXQ combined together?

My concern was predominantly about performance yes, I agree we could separate out the request body, we could even do it using a sequence just like the EXPath HTTP Client module does, but we would have to figure multipart into it in a better way, also there is still quite some overhead if we just convert the HTTP headers in/out of XML. I think HTTP should be fast and this overhead is unnecessary most of the time, hence me wondering whether implementations could short-cut it as an optimisation. I agree that lazy-evaluation may help, but again thats just another form of implementation specific optimisation.

[ChristianGruen]

Just a quick thought: maybe we could provide a %restxq:request("{$xyz}") annotation in order to bind an XML representation of all request information to a variable?

Regarding performance, I have similar concerns as Adam does, and I like that the current module is very light-weight and should hardly introduce any overhead in most implementations. However, I confess that I also haven’t thought about multipart requests yet. Maybe it’s still too early, as long as RESTXQ doesn’t support it?

[adamretter]

I had hoped to avoid something like %restxq:request, just as I have been dodging a @context equivalent in RESTXQ. I had rather hoped we could solve the issue in a Request Module instead.

Christian, with the model functions I proposed, would it be possible for you to optimise that in BaseX so you could shortcut the XML creation if the model function was used inside another Request module function? I think we should be able to do it in eXist. However it would be nice if I could somehow define this in standard XQuery terms, rather than having a note for implementers.

[msokolov]

On 1/5/2013 9:33 AM, Adam Retter wrote:

I am not sure of the need for url rewriting or parameter aliasing, do
you have a use-case that cannot be met by a Request Module and RESTXQ
combined together?

Well I have to admit I haven't thought very deeply about this, and am
just now familiarizing myself with the RESTXQ spec. But my idea was to
provide something like the ability to wrap an HttpServletRequest in the
Java servlet API when forwarding the request. RESTXQ seems to provide
most of what you want, but it is a declarative API. Suppose I want to
do something more functional like:

I have an app server that supports a different XML format for
http:request, and I'd like to provide that as a parameter.

I want to take all incoming requests, prefix their parameter names with
something (to guarantee uniqueness), and then add a few of my own
parameters drawn from a configuration file?

I want to decrypt the character data in all of the request parameters

Basically - anything that can operate on the request parameter map (or
headers) in a programmatic, rather than a declarative way. Maybe RESTXQ
does support that, and I just didn't understand correctly.

-Mike

[ChristianGruen]

@adamretter: yes, I think this surely could be done in some way. On the other hand, if we keep the existing functions of the module alive, the model function would only be necessary if a user really wants to retrieve the whole model, right?

[adamretter]

@msokolov I think the difference is that RESTXQ is the consumer, if you want to forward parameters you parse them in RESTXQ and then call your XQuery Module functions that you have written. There is no concept of Servlets in RESTXQ or forwarding to a request chain, because both of these are Java concepts which is out of scope.

RESTXQ is indeed declarative, so what happens if you need to programatically access things. The idea is that you would use RESTXQ with this Request Module to enable you to do exactly that.

[adamretter]

@ChristianGruen Well the existing functions could stay, and we say that they operate on a model which is implicitly passed in in the context of the module, however it felt cleaner and more FP to pass it in explicitly. The idea of having the model, is that the functions could operate on the model, or they could operate on a version of it passed in as XML if you were doing work with EXPath Request Module.

[ChristianGruen]

If we ignore the performance aspects, I wonder if we’d really need the new request:parameter(...) function? If we get back an XML fragment/document, we could use simple XPath to access all parameters. What do you think?

[j-maly]

Hi,
I will not comment on performance or implementation issues, because I have not studied code neither of exist nor baseX. Also the discussion has moved quite far now...

I think it should be possible to read all request parameters in a restxq annotated function - one way or another. I did try request module before creating the issue, but with no success (in eXist).

If the additional parameters are to be exposed using some functions (such as request:parameter-names etc.) - OK.
However, what I suggested would mean that the interface would remain abstracted from the HTTP.
There is, I think, a big difference between someone (me) who wants "just" to read request parameters and someone who needs some additional information from the request (headers etc.) and would benefit from having the access to the whole model of the request (exposed either as an XML document or via functional API).

It seems to me that the second use case deserves to be separated from my feature request.

[adamretter]

@ChristianGruen thats interesting as its kinda the inverse of what I was thinking of with the XML model, which makes me wonder if what I was thinking of is really required.

The idea of creating an XML model all of the time is a horrible one IMHO. It means you have a performance issue, and then you need to post process it to get the things you want from the model. The model was really just a device to allow better interop with the existing EXPath HTTP Client Module, I had hoped that clever implementers would create an optimisation to short-circuit the generation of the model, if the user put that function into another Request Module function, otherwise you have to create it as they may want to path-step it. But I see that having the two approaches available, could just serve to confuse users...

[adamretter]

@j-maly I am afraid the two things are linked. You want to read request-parameters, so we need a mechanism for that, I dont think that mechanism is an annotation because the annotations should be declarative and explicit, I think you need functions like request:parameter-names() and request:parameter(). I would love to deliver those functions for you, but firstly I have to figure out how to get multipart into the function spec, and that is my first concern. Most of this discussion so far has been about the part under '--- for later---', which as you can is kinda separate thing...

[ChristianGruen]

@adamretter: good to know, thanks. In our implementation, there is currently no link between RESTXQ and the EXPath HTTP Client Module, and no link between the Request and HTTP Module; but this could change if it seems an obvious path to follow in future.

[adamretter]

Jakub, I have now implemented the EXQuery Request Module 1.0 spec in eXist (as of revision 18206) and you can use that from inside RESTXQ functions. The only difference is that for the moment you will have to use the prefix "req:" and not "request:" as eXist current has a bug with multiple modules with the same prefix at the moment - we will fix that later.