Use lua-http for http operations if installed

[daurnimator]

This pull-request will make luarocks use lua-http for HTTP(S) operations if it is installed.

[daurnimator]

Anyone able to review this?

[mpeterv]

file:close() should be added here, or else the file can be used before all contents are saved.

[daurnimator]

resolved.

[mpeterv]

@daurnimator the new code is not covered because lua-http isn't installed on travis-ci =(

[mpeterv]

Is lua-http interface going to be as stable as luasocket or maybe a version check should be added here?

[hishamhm]

This is a lot of code and each module-backend option we add multiplies the complexity of our test matrix.

Supporting two implementations of the APIs in the codebase (one based on external commands, one based on Lua modules) is already a constant source of headaches, but we currently feel we have to do this because of bootstrapping.

So, the pressing question is: why add yet another code path? Adding this means supporting this in the long-term, having to track new dependencies, and making sure every feature we have or add now would need to work properly on all these new scenarios.

[daurnimator]

@daurnimator the new code is not covered because lua-http isn't installed on travis-ci =(

Indeed. I wasn't sure how best to add it to the testing matrix.
Normally I would add to the travis matrix (e.g. add a new variable "LUA_HTTP=true"); but luarocks doesn't seem to be doing that for anything else?
How is the curl fallback tested vs luasocket?

Is lua-http interface going to be as stable as luasocket or maybe a version check should be added here?

The lua-http interfaces I've used I consider stable, minus some of the error indications.
I do need to create a versioning scheme for lua-http though.

[daurnimator]

So, the pressing question is: why add yet another code path? Adding this means supporting this in the long-term, having to track new dependencies, and making sure every feature we have or add now would need to work properly on all these new scenarios.

I find the luasocket http/luasec https modules a mess; I created lua-http as a breath of fresh air.
Why prefer lua-http?:

• Redirects are supported well (luasocket doesn't redirect from http => https; no strange interactions between retries)
• SSL supported in the one interface
• HTTP2 is supported
• Able to be async: in future this means we could download packages in parallel

[hishamhm]

You only answered why you like lua-http better than luasocket (which is obvious, since it's your module).

Unfortunately, you didn't provide a convincing argument as for what what real, current problems does this fix to the point that it would justify the additional complexities I enumerated above.

One alternative way to go about this would be to think how could we make the structure of LuaRocks more modular to accomodate alternative implementations of things. This, I think, would spark a wider architectural discussion.

[Tieske]

I think @hishamhm has a good point; we need really, really convincing arguments to add another set of complexities

[hishamhm]

@daurnimator Revisiting this, there is one killer feature which I think would be worth adding a lua-http backend to luarocks.fs: HTTP persistent connections. Running a command like luarocks install busted on a blank system opens a lot of connections to luarocks.org and it seems a noticeable amount of time is spent on that. I did change a bunch of wget calls in the test suite into one multi-file wget call and got nice speedups. Would lua-http be able to that kind of speed up for LuaRocks as well?

[daurnimator]

@hishamhm if I get time to finish off the connection pooling stuff it could. All the foundational stuff is there for it, just need to add the actual pool.