Cannot make URL query string with a parameter without a value

[agilevic]

URL query string may contain a parameter, which has no value i.e. http://host/path/?foo or http://host/path/?a=1&foo. Currently Requests does not provide support for that.

In [68]: d
Out[68]: {'a': 1, 'foo': None}

In [69]: tl
Out[69]: [('a', 1), ('foo',)]

In [70]: RequestEncodingMixin._encode_params(d)
Out[70]: 'a=1'

In [71]: RequestEncodingMixin._encode_params(tl)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-71-5d4dac855108> in <module>()
----> 1 RequestEncodingMixin._encode_params(tl)

/home/f557010/jpm/local/lib/python2.7/site-packages/requests/models.pyc in _encode_params(data)
     87         elif hasattr(data, '__iter__'):
     88             result = []
---> 89             for k, vs in to_key_val_list(data):
     90                 if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):
     91                     vs = [vs]

ValueError: need more than 1 value to unpack

Expected:

'a=1&foo'

[Lukasa]

I can see some value in this. For API reasons it could only ever work with the 'list of tuples' approach, but I'd be ok with us adding support for this. @sigmavirus24?

[sigmavirus24]

I don't think we use it (yet) but it seems that urllib3 uses urlencode and so do we

If we examine how that behaves, you can see that it doesn't like either of the proposed ways of working with this. {'foo': None} will "work" but does not do the right thing. This is probably why we avoided this before. That said, RFC 3986 has a very ... loose definition of the query part of a URI, so in my opinion we should handle it. That said, I'm not sure there's any tools to readily allow us to handle it. =/

>>> u = urlparse.urlparse('http://example.com/foo?bar')
>>> u
ParseResult(scheme='http', netloc='example.com', path='/foo', params='', query='bar', fragment='')
>>> urlparse.parse_qs(u.query)
{}
>>> urllib.urlencode({'foo': None})
'foo=None'
>>> urllib.urlencode([('foo',)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1336, in urlencode
    for k, v in query:
ValueError: need more than 1 value to unpack

[piotr-dobrogost]

Related: "urlencode of a None value uses the string 'None'" – https://bugs.python.org/issue18857

[Lukasa]

Give @piotr-dobrogost's input, @agilevic have you tried using the empty string as the value there? Does it work with your API server?

[Lukasa]

Bump. =)

[sigmavirus24]

>>> import requests
>>> r = requests.get('https://httpbin.org/get', params={'foo': ''})
>>> r.request.url
'https://httpbin.org/get?foo='

[agilevic]

It's not the same as having a parameter without a value. Yours renders the = sign after the parameter name. Some applications will work, but as a matter of providing a complete solution this exact case should be addressed. That urllib doesn't do it is of no significance. Requests does many things better than standard libraries for HTTP - that is its reason to exist.

[Lukasa]

@agilevic What would your proposed API design for this feature be?

[frnhr]

Here is a crazy thought:

>>> import requests
>>> r = requests.get('https://httpbin.org/get', params={'foo': None})
>>> r.request.url
'https://httpbin.org/get?foo'

That is what I think should be happening.

What is actually happening:

'https://httpbin.org/get'

:(

[Lukasa]

@frnhr So the reason that doesn't work with our API is that setting a key to None is the signal for "please remove this key from the map". We have that signal because some parameters can be persisted on a Session object itself, and users occasionally want to be able to suppress those parameters on a per-request basis.

[frnhr]

I'm afraid I don't see what the session object has to do with this bit of API, sincerely. But ok, maybeFalse then, or even some SpecialImportableObject instead of None?

On 17 Sep 2016, at 07:47, Cory Benfield notifications@github.com wrote:

@frnhr So the reason that doesn't work with our API is that setting a key to None is the signal for "please remove this key from the map". We have that signal because some parameters can be persisted on a Session object itself, and users occasionally want to be able to suppress those parameters on a per-request basis.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Lukasa]

@frnhr The Session API is relevant because the requests. API is built on top of the Session API: it's a subset of that functionality, a convenience wrapper.

So we certainly could do it, but I'm not sure to what extent it's worthwhile. When not using key-value mapping, you should just pass a string to the params field: params="foo".

[whitej125]

What if we were to make a specific typed argument that could indicate to requests that the param is to be added without a value?

# defined somewhere in requests
class ValuelessParam(object):
    pass
....
....

params = {'foo': 'some_param', 'bar': requests.ValuelessParam()}

requests.get('http://something', params=params)

# url should be 'http://something?foo=some_param&bar'

Its not None, its not a scalar constant... so it should be backwards compatible. Under the hood... we could check for that value type and specially append this parameter to the constructed url.

[Lukasa]

So while that will definitely work, I don't think that API will get past Kenneth.

[kennethreitz]

Yeah, No. I'd rather support a built-in like None, and I'm not sure that's the best idea — but it could work well. False would not.

[Lukasa]

None won't work either: the codebase already gives it the meaning of "unset the value set at the Session level.

[kennethreitz]

That would be a pretty major change though, and I don't think it would benefit many people. Perhaps an empty tuple could be considered. (e.g. (,).

[nellson-CSC]

Not sure if this was resolved, but I found this thread trying to do the exact thing of adding a key with out a value. "QueryAll" in my case, but I have had a number of causes in my RestAPI Automation to make use of this kind of function.

[kennethreitz]

what happens if you pass {'QueryAll': ''}?

[nellson-CSC]

I get &QueryAll= at the end.

[nellson-CSC]

Looks like it's kinda up to the API on how it handles the open "=" then, PasswordState's API took it as long as it was at the end of my params list, if I moved it to the beginning it error-ed.

[kennethreitz]

in 3.0, i think we can consider making emptry string not result in an =

[nellson-CSC]

That would be great :-D So far my Passwordstate project can move forwward with the QueryAll= format at the end of teh params string so I am back on track. I'll watch this thread :-D

Thanks Kenneth!

[nellson-CSC]

Have not heard or seen anything yet, but have not circled back around to see if any updates had been released. Nick From: Alex Zagoro <notifications@github.com> Sent: Saturday, September 22, 2018 10:19 AM To: requests/requests <requests@noreply.github.com> Cc: Ellson, Nick <NEllson@columbia.com>; Comment <comment@noreply.github.com> Subject: Re: [requests/requests] Cannot make URL query string with a parameter without a value (#2651) heya, any updates on this?

[roncrdb]

I'm running into this problem at the moment as well. Has anyone else found any way around this problem?

[samuraii]

I believe there is still no solution in requests for this simple, but very common feature. weird.

[gmcabiato]

Unfortunately facing the same issue.
Wouldn't it be a better pattern or option to be able to inject/pass an optional custom formatter/encoder or a specific flag for handling None behaviour?

@Lukasa You seem to know the codebase better than most of us, what do you think about it?

Wouldn't that resolve in a non-breaking-change, @kennethreitz? Defaults could be set to mimic 2.x behaviour.

[flisz]

It appears that this is still adding the = when an empty string is passed in. Time to bang out a work-around!

[bwilgus]

So I'm guessing this is still a problem?

[joewesch]

If you have parameters that have no value, you can list them as part of the url and any additional parameters will append with & instead of starting with ?:

In [3]: r = requests.get("http://www.google.com", params={'test': 'true'})

In [4]: r.url
Out[4]: 'http://www.google.com/?test=true'

In [5]: r = requests.get("http://www.google.com?test2", params={'test': 'true'})

In [6]: r.url
Out[6]: 'http://www.google.com/?test2&test=true'

[KES777]

How about this rule?

{ key: undefined } to ? (not included)
{ key: null } to ?key
{ key: '' } to ?key=
{ key: 'null' } to ?key=null

[VeNoMouS]

2020... and still an issue.... SMH

[idkq]

What's the solution folks?