Add support for unix domain sockets #276

msabramo · 2014-11-25T02:29:22Z

To issue HTTP requests to a UNIX domain socket:

Use new "http+unix://" scheme in URL.
Use urlencoded socket path as the hostname part of the URL.
e.g.: for a socket at /tmp/profilesvc.sock, you could do:
```
http http+unix://%2Ftmp%2Fprofilesvc.sock/status/pid
```

This uses https://pypi.python.org/pypi/requests-unixsocket

Screenshot

Cc: @shin-, @jakubroztocil, @monsanto, @np, @nuxlli, @matrixise, @remmelt

msabramo · 2014-11-25T05:08:25Z

The Travis CI test failure is a bit of a puzzlement, because my tests aren't failing for me locally on OS X or Ubuntu 14.04.

msabramo · 2014-11-25T07:34:24Z

Oh, I think I see what the problem might be. It's a race condition, because I don't wait for the waitress server to be launched before trying to hit it. Often locally the server happens to come up in time, but for some reason on Travis CI, the test executes before waitress has started serving.

msabramo · 2014-11-25T08:02:35Z

OK, test failures fixed. There are some other test failures that are not related to the code that I added/changed.

jkbrzt · 2014-11-25T08:21:35Z

httpie/adapters.py

+        netloc is a percent-encoded path to a unix domain socket. E.g.:
+        'http+unix://%2Ftmp%2Fprofilesvc.sock/status/pid'
+        """
+        HTTPConnection.__init__(self, 'localhost', timeout=timeout)


Why not super()?

I personally don't use super a ton, because I find it to not be an easier or DRYer than explicitly calling the superclass. This kind of sums it up for me:

http://nedbatchelder.com/blog/200709/pythons_super_considered_harmful.html

Also for some stupid reason, even after years of Python programming, I still sometimes forget the order of arguments -- is it (DerivedClass, self) or ('self, DerivedClass)? (It's the former but I've screwed it up many times) :-)

That said if this code is going to be merged into your project and you prefer super, I'm totally willing to change it.

I see your points. However, for the sake of consistency with the rest of the codebase, I'd still prefer super().

I definitely would be willing to do that, but I think a bunch of the code is going to move to a separate module and so this will become a moot point.

jkbrzt · 2014-11-25T08:25:03Z

This is really great. Thanks @msabramo!

A couple of things:

Please add yourself to AUTHORS
Update CHANGELOG
Document the new socket support in README

msabramo · 2014-11-25T14:41:39Z

You're welcome! Thanks for httpie!

I'll do the things you said (including using super if that's what you prefer).

By the way, I'm toying with the idea of putting this code for using requests with unix sockets as a module in PyPI. That way it could be leveraged by multiple projects. Would you rather wait for that? Alternatively, you could merge as-is and we can extract later.

jkbrzt · 2014-11-25T22:31:10Z

httpie/adapters.py

+import socket
+
+from requests.adapters import HTTPAdapter
+from requests.compat import urlparse, unquote


Btw, there is also httpie.compat and it's what is used elsewhere in HTTPie. Could you use it as a proxy to import these two functions? Basically it just needs to be added here: https://github.com/msabramo/httpie/blob/1b13eb3af97fac71276e9af29e8c5d8c883234a1/httpie/compat.py#L7

I think this and a few of your other points will become moot when we use the separate module from PyPI.

jkbrzt · 2014-11-25T22:42:59Z

I think it's definitely a good idea to put it on PyPi as a standalone package.

Then I could modify the plugin API to make it possible to write adapter plugins like this one, and another small package could expose this functionality as a plugin for HTTPie.

monsanto · 2014-11-26T04:10:51Z

@msabramo It would be awesome if you published that module to PyPI. And thanks for putting in this work!

msabramo · 2014-11-26T05:12:02Z

@monsanto: Yeah that's my plan. Then httpie and other projects like docker-py could probably use it.

msabramo · 2014-11-26T08:05:14Z

Here's a preview of where I was going:

https://github.com/msabramo/requests-unixsocket

I want to add Travis CI and some better docs and possibly some function you could call to monkeypatch the default requests.Session so requests.get would work.

If folks have thoughts, feel free to send PRs or file issues.

msabramo · 2014-11-28T07:53:02Z

@jakubroztocil, @monsanto: I just pushed my module to PyPI -- take a look:

https://pypi.python.org/pypi/requests-unixsocket/0.1.1

I'll try later to change this PR to use that module and address @jakubroztocil's other comments.

msabramo · 2014-11-28T18:42:09Z

Please add yourself to AUTHORS

You already added me in a1682d0

Update CHANGELOG

OK. (took me a minute to realize there is no CHANGELOG file; it's just a section in README.rst -- I wonder if it should be extracted out into a separate file?)

Document the new socket support in README

OK.

msabramo · 2014-11-28T19:12:04Z

The test failures look like intermittent failures unrelated to my change; probably due to the dependence on httpbin. I'm thinking it might be useful to use something like pytest-httpbin.

msabramo · 2014-11-28T19:16:45Z

Oops, it's already using pytest-httpbin

msabramo · 2014-12-04T01:40:51Z

@jakubroztocil: I think this is ready for a second round of review when you're ready.

msabramo · 2014-12-12T07:58:07Z

Bump

jkbrzt · 2014-12-12T08:10:02Z

@msabramo Unfortunately I'm recovering from an injury. Will look into this as soon as possible.

msabramo · 2014-12-12T14:29:59Z

Sorry to hear that!

Best wishes for a speedy recovery!

jkbrzt · 2015-02-01T14:15:30Z

@msabramo what do you think about adding this as a plugin (#276 (comment))? There is also another plugin candidate that is basically a requests adapter (#298).

msabramo · 2015-02-04T16:04:52Z

I'm open to creating an HTTPie plugin. I need some guidance though.

From a glance at the code, it looks like there are 3 types of plugins:

auth (the only kind that is documented in README.rst by the way)
converter (undocumented)
formattter (undocumented)

If I am not mistaken, these all get called after the request has already completed. In that case that would be too late to try to open a unix socket.

Does there need to be a new type of plugin perhaps -- a connection plugin?

To issue HTTP requests to a UNIX domain socket: * Use new "http+unix://" scheme in URL. * Use urlencoded socket path as the hostname part of the URL. e.g.: for a socket at /tmp/profilesvc.sock, you could do: http http+unix://%2Ftmp%2Fprofilesvc.sock/status/pid This uses https://pypi.python.org/pypi/requests-unixsocket Fixes: httpieGH-209 (httpie#209)

msabramo · 2015-02-04T16:44:36Z

Actually it's looking like I can maybe make it an auth plugin and then register my requests adapter at the top level of the module so that it executes at import time. But this feels a little hacky so maybe a connection plugin is still a good idea.

- if not (self.args.url.startswith((HTTP, HTTPS))): + if not re.match(r'^http.*://', self.args.url): This allows us to accommodate new connection schemes such as the "http+unix" scheme being discussed in httpie#276

msabramo · 2015-02-04T17:19:51Z

@jakubroztocil: Please see #299

With that, I can get an auth plugin to work.

msabramo · 2015-02-04T18:23:35Z

And here's the unix socket "auth plugin" I hacked together:

https://github.com/msabramo/httpie-unixsocket

I guess it would be nice if it were a "connection plugin" rather than an "auth plugin", but we'd have to add support to HTTPie for that.

Closes #299 See also #276

jkbrzt · 2015-02-05T13:46:19Z

@msabramo

From a glance at the code, it looks like there are 3 types of plugins:

auth (the only kind that is documented in README.rst by the way)

converter (undocumented)

formattter (undocumented)

That is correct. The plugin API is still work in progress and will probably be completely reworked by v1.0.0, that's why it's lacking docs. But for now, I will add a new plugin type, an AdapterPlugin (python-requests: transport adapters). HTTPie will load these plugins on startup and mount them (just like you did in the first version of this pull request).

#276, #298

httpie/cli#276

jkbrzt · 2015-02-05T14:30:08Z

@msabramo Let me please know if it's okay. I'll then release v0.9.1 and you can publish the plugin on PyPi.

jkbrzt · 2015-02-07T16:14:57Z

@msabramo HTTPie v0.9.1 with transport adapter plugin support is out. So the plugin can be released on PyPi as well (after httpie/httpie-unixsocket#3 has been merged).

jkbrzt · 2015-02-07T16:50:08Z

Implemented as a plugin supported in HTTPie v0.9.1+: https://github.com/msabramo/httpie-unixsocket

msabramo mentioned this pull request Nov 25, 2014

support http over unix domain sockets #209

Closed

msabramo force-pushed the unix_domain_sockets branch 2 times, most recently from de27129 to fec497a Compare November 25, 2014 03:24

msabramo force-pushed the unix_domain_sockets branch from fec497a to 666d7e5 Compare November 25, 2014 07:30

msabramo force-pushed the unix_domain_sockets branch from 666d7e5 to 1b13eb3 Compare November 25, 2014 07:45

jkbrzt reviewed Nov 25, 2014
View reviewed changes

msabramo force-pushed the unix_domain_sockets branch 3 times, most recently from f6e78dc to f636886 Compare November 28, 2014 18:41

msabramo force-pushed the unix_domain_sockets branch from f636886 to 9d5d55c Compare November 28, 2014 18:53

msabramo force-pushed the unix_domain_sockets branch from 9d5d55c to 153412c Compare December 4, 2014 01:57

msabramo force-pushed the unix_domain_sockets branch from 153412c to e72b6ab Compare February 4, 2015 16:12

msabramo mentioned this pull request Feb 4, 2015

Loosen check for "http" scheme in URL #299

Closed

jkbrzt added a commit that referenced this pull request Feb 5, 2015

Allow custom URL schemes

b125ce5

Closes #299 See also #276

jkbrzt added a commit that referenced this pull request Feb 5, 2015

Added support for transport adapter plugins

687a6a7

#276, #298

jkbrzt added a commit to jkbrzt/httpie-unixsocket that referenced this pull request Feb 5, 2015

Updated to use the new TransportPlugin API.

5915b38

httpie/cli#276

jkbrzt mentioned this pull request Feb 5, 2015

Updated to use the new TransportPlugin API. httpie/httpie-unixsocket#3

Merged

jkbrzt closed this Feb 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for unix domain sockets #276

Add support for unix domain sockets #276

msabramo commented Nov 25, 2014

msabramo commented Nov 25, 2014

msabramo commented Nov 25, 2014

msabramo commented Nov 25, 2014

jkbrzt Nov 25, 2014

msabramo Nov 25, 2014

jkbrzt Nov 25, 2014

msabramo Nov 28, 2014

jkbrzt commented Nov 25, 2014

msabramo commented Nov 25, 2014

jkbrzt Nov 25, 2014

msabramo Nov 28, 2014

jkbrzt commented Nov 25, 2014

monsanto commented Nov 26, 2014

msabramo commented Nov 26, 2014

msabramo commented Nov 26, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Dec 4, 2014

msabramo commented Dec 12, 2014

jkbrzt commented Dec 12, 2014

msabramo commented Dec 12, 2014

jkbrzt commented Feb 1, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

jkbrzt commented Feb 5, 2015

jkbrzt commented Feb 5, 2015

jkbrzt commented Feb 7, 2015

jkbrzt commented Feb 7, 2015

Add support for unix domain sockets #276

Add support for unix domain sockets #276

Conversation

msabramo commented Nov 25, 2014

Screenshot

msabramo commented Nov 25, 2014

msabramo commented Nov 25, 2014

msabramo commented Nov 25, 2014

jkbrzt Nov 25, 2014

Choose a reason for hiding this comment

msabramo Nov 25, 2014

Choose a reason for hiding this comment

jkbrzt Nov 25, 2014

Choose a reason for hiding this comment

msabramo Nov 28, 2014

Choose a reason for hiding this comment

jkbrzt commented Nov 25, 2014

msabramo commented Nov 25, 2014

jkbrzt Nov 25, 2014

Choose a reason for hiding this comment

msabramo Nov 28, 2014

Choose a reason for hiding this comment

jkbrzt commented Nov 25, 2014

monsanto commented Nov 26, 2014

msabramo commented Nov 26, 2014

msabramo commented Nov 26, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Nov 28, 2014

msabramo commented Dec 4, 2014

msabramo commented Dec 12, 2014

jkbrzt commented Dec 12, 2014

msabramo commented Dec 12, 2014

jkbrzt commented Feb 1, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

msabramo commented Feb 4, 2015

jkbrzt commented Feb 5, 2015

jkbrzt commented Feb 5, 2015

jkbrzt commented Feb 7, 2015

jkbrzt commented Feb 7, 2015