boto.exception.BotoServerError: BotoServerError: 503 Service Unavailable

[krismarc]

Hi boto community,

if proxy is configured in boto config file the script is checking OS for proxy configuration anyway and in some cases like SLES the no_proxy variable values are 'comma + space' separated. This results in skip_proxy with false.

[Boto]
debug = 2
num_retries = 1
proxy = random.proxy.com
proxy_port = 8080

Added some prints to debug:

True
path /
auth_path /
path2 /
use_proxy
True
**skip_proxy
False**
path3 https://<path changed>/
method:(GET) protocol:(https) host(random.host.com) port(443) path(https://<path changed>/) params({}) headers({}) body()
reply: 'HTTP/1.1 200 Connection established\r\n'
send: 'GET https://<path changed>/ HTTP/1.1\r\nHost: **** \r\nAccept-Encoding: identity\r\nDate: Thu, 05 Dec 2019 13:15:05 GMT\r\nContent-Length: 0\r\nAuthorization: AWS 2*** \r\nUser-Agent: Boto/2.49.0 Python/2.7.13 Linux/4.12.14-95.24-default\r\n\r\n'
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
header: Server: squid
header: Mime-Version: 1.0
header: Date: Thu, 05 Dec 2019 13:15:05 GMT
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 18387
header: X-Squid-Error: ERR_DNS_FAIL 0
header: X-Cache: MISS from swre-prx001-ch-her-4
header: Via: 1.1 swre-prx001-ch-her-4 (squid)
header: Connection: close
send: 'GET https://<path changed>/ HTTP/1.1\r\nHost: **** \r\nAccept-Encoding: identity\r\nDate: Thu, 05 Dec 2019 13:15:05 GMT\r\nContent-Length: 0\r\nAuthorization: AWS 2*** \r\nUser-Agent: Boto/2.49.0 Python/2.7.13 Linux/4.12.14-95.24-default\r\n\r\n'
reply: ''
reply: 'HTTP/1.1 200 Connection established\r\n'
Traceback (most recent call last):
  File "s3.py", line 4, in <module>
    for bucket in conn.get_all_buckets():
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 443, in get_all_buckets
    response = self.make_request('GET', headers=headers)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 675, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1080, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
    raise BotoServerError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 503 Service Unavailable

This starts here: (fine only if no_proxy values are just comma separated)

boto/boto/connection.py

Line 720 in 91ba037

for name in self.no_proxy.split(','):

working ("just comma separated"):

cat /etc/sysconfig/proxy | grep NO_PROXY
Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost,127.0.0.1,.expected.domain.com"

not working ("comma + space" separated):

cat /etc/sysconfig/proxy | grep NO_PROXY
Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1, .expected.domain.com"

If I change the above code to (with extra space for split) then it starts to work in opposite way:
for name in self.no_proxy.split(', '):

Kind regards,
KrzMar

[krismarc]

me@host:~> cat s3.py
import boto
import boto.s3.connection
conn = boto.connect_s3(
       aws_access_key_id = '***',
       aws_secret_access_key = '***',
       host = '***',
       calling_format = boto.s3.connection.OrdinaryCallingFormat())

for bucket in conn.get_all_buckets():
    print (bucket.name, bucket.creation_date)

Better example:

me@host:~> export no_proxy='localhost,127.0.0.1,.randomdomain.com' ; python s3.py
#bucket: test   2019-11-29T16:00:57.532Z

me@host:~> export no_proxy='localhost, 127.0.0.1, .randomdomain.com' ;  python s3.py
Traceback (most recent call last):
  File "s3.py", line 4, in <module>
    for bucket in conn.get_all_buckets():
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 443, in get_all_buckets
    response = self.make_request('GET', headers=headers)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 671, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1028, in _mexe
    raise BotoServerError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 503 Service Unavailable