Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Piped URL to httpie with xargs creates a POST request unexpectedly #1017

Closed
webdog opened this issue Jan 12, 2021 · 3 comments
Closed

Piped URL to httpie with xargs creates a POST request unexpectedly #1017

webdog opened this issue Jan 12, 2021 · 3 comments

Comments

@webdog
Copy link

webdog commented Jan 12, 2021

Hello!

Thank you for this utility! Has made requesting/testing URLs in development very handy for me.

I came across a bug that was resolved by using the instructions in #150 and #660 , but I am seeing unexpected behavior in the request itself that I couldn't find in reported in another issue.

When parsing a URL from a pipe, I expected httpie to create a GET request to the specified URL:

http https://api.weather.gov/points/41.9752,-87.9035 | jq '.properties.forecast | tostring' | sed 's/"//g' | xargs http

Translates to: http https://api.weather.gov/gridpoints/LOT/65,76/forecast

When I run the piped request, the URL returns 403 Forbidden, indicating an unauthorized request. Running the request with debug indicates I've sent a POST method to the URL (Refer to requests.request section)

❯ http  https://api.weather.gov/points/41.9752,-87.9035 | jq '.properties.forecast | tostring' | sed 's/"//g' | xargs http --debug                                                                                                      
HTTPie 2.2.0
Requests 2.24.0
Pygments 2.6.1
Python 3.9.0 (default, Dec  6 2020, 18:02:34) 
[Clang 12.0.0 (clang-1200.0.32.27)]
/usr/local/Cellar/httpie/2.2.0_1/libexec/bin/python
Darwin 19.6.0

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/Users/cwebdog/.config/httpie'),
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x103ad0700>,
 'program_name': 'http',
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': False,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

>>> requests.request(**{'auth': None,
 'data': b'',
 'files': RequestFilesDict(),
 'headers': {'User-Agent': b'HTTPie/2.2.0'},
 'method': 'post',
 'params': RequestQueryParamsDict(),
 'url': 'https://api.weather.gov/gridpoints/LOT/65,76/forecast'})

HTTP/1.1 403 Forbidden
Cache-Control: max-age=0
Connection: close
Content-Length: 309
Content-Type: text/html
Date: Tue, 12 Jan 2021 16:42:18 GMT
Expires: Tue, 12 Jan 2021 16:42:18 GMT
Mime-Version: 1.0
Server: AkamaiGHost
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
Vary: Accept,Feature-Flags,Accept-Language
X-Edge-Request-ID: 3d0c2d7

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;api&#46;weather&#46;gov&#47;gridpoints&#47;LOT&#47;65&#44;76&#47;forecast" on this server.<P>
Reference&#32;&#35;18&#46;6fe13217&#46;1610469738&#46;3d0c2d7
</BODY>
</HTML>

By appending --ignore-stdin to the request URL, or by declaring GET as the method in the first argument following http maintains the request as a GET:

❯ http  https://api.weather.gov/points/41.9752,-87.9035 | jq '.properties.forecast | tostring' | sed 's/"//g' | xargs http --ignore-stdin --debug                                                                                         
HTTPie 2.2.0
Requests 2.24.0
Pygments 2.6.1
Python 3.9.0 (default, Dec  6 2020, 18:02:34) 
[Clang 12.0.0 (clang-1200.0.32.27)]
/usr/local/Cellar/httpie/2.2.0_1/libexec/bin/python
Darwin 19.6.0

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/Users/cwebdog/.config/httpie'),
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x1025a2700>,
 'program_name': 'http',
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': False,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'files': RequestFilesDict(),
 'headers': {'User-Agent': b'HTTPie/2.2.0'},
 'method': 'get',
 'params': RequestQueryParamsDict(),
 'url': 'https://api.weather.gov/gridpoints/LOT/65,76/forecast'})

HTTP/1.1 200 OK
Access-Control-Allow-Headers: Feature-Flags
Access-Control-Allow-Origin: *
Cache-Control: public, max-age=842, s-maxage=3600
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 1514
Content-Type: application/geo+json
Date: Tue, 12 Jan 2021 16:56:32 GMT
Expires: Tue, 12 Jan 2021 17:10:34 GMT
Last-Modified: Sun, 26 Dec 2021 06:21:50 GMT
Server: nginx/1.16.1
Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
Vary: Accept-Encoding
Vary: Accept,Feature-Flags,Accept-Language
X-Correlation-ID: 8942acf
X-Edge-Request-ID: 84a76f69
X-Request-ID: c9da5813-06a6-4b4d-ab6a-1789ae353037
X-Server-ID: vm-lnx-nids-apiapp3.ncep.noaa.gov
@jkbrzt
Copy link
Member

jkbrzt commented Jan 12, 2021

Hi @webdog! xargs doesn’t use /dev/tty for the child’s stdin by default. Therefore, HTTPie will read data from stdin and assume you’re trying to send some data (even if it’s just '' — this might change in a future version), and default to a POST request.

The most straightforward solution is to use the already mentioned --ignore-stdin even here:

$ echo pie.dev/get | xargs http --ignore-stdin

Another option would be to use xargs -o:

$ echo pie.dev/get | xargs -o http 
$ man xargs
  …
   -o      Reopen stdin as /dev/tty in the child process before executing the command.
             This is useful if you want xargs to run an interactive application.
  …

@webdog
Copy link
Author

webdog commented Jan 12, 2021

@jakubroztocil Today I learned! That makes a lot of sense. Thank you for the quick response!

@jkbrzt
Copy link
Member

jkbrzt commented Jan 12, 2021

Happy to help! 🥧

@jkbrzt jkbrzt closed this as completed Jan 12, 2021
@landure landure mentioned this issue Aug 29, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jkbrzt @webdog