Download of less.tar.gz decompresses with wget/curl on Linux

[ermel272]

The download of the less tarball seems to intermittently decompress on CentOS & Alpine linux using both wget and curl:

[wsse-231-50 ~]# curl http://www.greenwoodsoftware.com/less/less-563.tar.gz --verbose --output less2.tar.gz
* About to connect() to proxy X.X.X.X port 8000 (#0)
*   Trying X.X.X.X...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to X.X.X.X (X.X.X.X) port 8000 (#0)
> GET http://www.greenwoodsoftware.com/less/less-563.tar.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.greenwoodsoftware.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Expires: Sun, 17 Oct 2021 13:46:51 GMT
< Date: Tue, 12 Oct 2021 13:46:51 GMT
< Server: Apache/2
< Last-Modified: Sun, 14 Jun 2020 03:35:46 GMT
< ETag: "51e94-5a80301722da7"
< Accept-Ranges: bytes
< Vary: User-Agent
< Content-Type: application/x-gzip
< Via: 1.1 usaai9102-prx002
< Age: 58
< Transfer-Encoding: chunked
< 
{ [data not shown]
100 1310k    0 1310k    0     0  4818k      0 --:--:-- --:--:-- --:--:-- 4833k
* Connection #0 to host X.X.X.X left intact
[wsse-231-50 ~]# gzip -d less2.tar.gz 

gzip: less2.tar.gz: not in gzip format
[wsse-231-50 ~]# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.44 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 
[wsse-231-50 ~]# gzip --version
gzip 1.5
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.

Note that X.X.X.X in the above snippet corresponds to the IP address of an internal proxy.

This issue is not reproducible on MacOS, configured to use the same proxy:

chrisermel@M-C02ZH9GYMD6N:~/Documents$ curl http://www.greenwoodsoftware.com/less/less-563.tar.gz --verbose --output less.tar.gz
* Uses proxy env variable no_proxy == 'localhost,local.home,127.0.0.0/8,X.0.0.0/8,10.0.0.0/8'
* Uses proxy env variable http_proxy == 'http://X.X.X.X:8000'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying X.X.X.X...
* TCP_NODELAY set
* Connected to X.X.X.X (X.X.X.X) port 8000 (#0)
> GET http://www.greenwoodsoftware.com/less/less-563.tar.gz HTTP/1.1
> Host: www.greenwoodsoftware.com
> User-Agent: curl/7.64.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Date: Tue, 12 Oct 2021 14:16:46 GMT
< Server: Apache/2
< Last-Modified: Sun, 14 Jun 2020 03:35:46 GMT
< ETag: "51e94-5a80301722da7"
< Accept-Ranges: bytes
< Content-Length: 335508
< Vary: User-Agent
< Content-Type: application/x-gzip
< Content-Encoding: x-gzip
< 
{ [1055 bytes data]
100  327k  100  327k    0     0   572k      0 --:--:-- --:--:-- --:--:--  572k
* Connection #0 to host X.X.X.X left intact
* Closing connection 0
chrisermel@M-C02ZH9GYMD6N:~/Documents$ gzip -d less.tar.gz 
chrisermel@M-C02ZH9GYMD6N:~/Documents$ curl --version
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
chrisermel@M-C02ZH9GYMD6N:~/Documents$ gzip --version
Apple gzip 287.100.2

[polluks]

My Debian says

$ file less2.tar.gz
less2.tar.gz: gzip compressed data, last modified: Sun Jun 14 03:34:59 2020, from Unix, original size 1341440

Maybe it's "Content-Encoding: x-gzip"?
See also https://stackoverflow.com/a/56008375/1430535

[ermel272]

Here's the weird thing... I reproduced it this morning with wget... You can see here that the first download successfully pulls the compressed tar:

[root@wsse-231-84 ~]# wget http://www.greenwoodsoftware.com/less/less-563.tar.gz --verbose
--2021-10-13 08:24:13--  http://www.greenwoodsoftware.com/less/less-563.tar.gz
Connecting to X.X.X.X... connected.
Proxy request sent, awaiting response... 200 OK
Length: 335508 (328K) [application/x-gzip]
Saving to: ‘less-563.tar.gz’

100%[=============================================================================================================================================================================================================================================>] 335,508     1.24MB/s   in 0.3s   

2021-10-13 08:24:14 (1.24 MB/s) - ‘less-563.tar.gz’ saved [335508/335508]

[wsse-231-84 ~]# file less-563.tar.gz 
less-563.tar.gz: gzip compressed data, from Unix, last modified: Sat Jun 13 23:34:59 2020
[wsse-231-84 ~]# rm less-563.tar.gz 
rm: remove regular file ‘less-563.tar.gz’? y

But then immediately after, downloading it again, file is now decompressed on download:

[wsse-231-84 ~]# wget http://www.greenwoodsoftware.com/less/less-563.tar.gz --verbose
--2021-10-13 08:24:36--  http://www.greenwoodsoftware.com/less/less-563.tar.gz
Connecting to X.X.X.X... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘less-563.tar.gz’

    [  <=>                                                                                                                                                                                                                                         ] 1,341,440   5.15MB/s   in 0.2s   

2021-10-13 08:24:36 (5.15 MB/s) - ‘less-563.tar.gz’ saved [1341440]

[wsse-231-84 ~]# file less-563.tar.gz 
less-563.tar.gz: POSIX tar archive (GNU)

Only thing different that I can see between the two is that the second one has unspecified length.

[gwsw]

This is the same as issue #22. I believe it is due to the Content-Encoding header. I have tried many suggestions of Apache configuration changes to eliminate this header but so far have been unsuccessful.

[gwsw]

Duplicate of #22