Binding a particular IP

[jibaku]

It would be great to be able to bound a particular IP as the source address (for hosts with multiple IP addresses).

[kennethreitz]

Interesting. How do you invision the usage?

[kennethreitz]

^ urllib3/urllib3#9

[jibaku]

The goal is to write a crawler using an IP associated with a particular country. But I don't know how it would be used on a API level.

[kennethreitz]

Couldn't you just send a request to an IP with a particular Host header?

[jibaku]

No, because it's to have the specific IP in the web server log (to be reversed to a particular country using tools like GeoIP).

[kennethreitz]

I don't follow.

>>> requests.get('http://184.106.67.239', headers={'host': 'ci.kennethreitz.com'})
<Response [200]>

[piotr-dobrogost]

import socket
import urllib2
true_socket = socket.socket

def make_bound_socket(source_ip):
    def bound_socket(*a, **k):
        sock = true_socket(*a, **k)
        sock.bind((source_ip, 0))
        return sock
    return bound_socket

socket.socket = make_bound_socket('<some source ip>')
print urllib2.urlopen('http://httpbin.org/ip').read()

socket.socket = make_bound_socket('<some other source ip>')
print urllib2.urlopen('http://httpbin.org/ip').read()

If traffic going out from the two source ip numbers goes out through two different public ip numbers one should see these different public numbers in the origin field of the response.

[kennethreitz]

I think this is out of scope.

[emidln]

For people interested in this still, you might want to take a look at urlib3's HttpConnectionPool, in particular where new httplib/http.client HttpConnections are spun up so you can pass in source_address. If you do that, then you can keep on exposing this in request's Request model, and use it as you wish.

[jessesherlock]

For those arriving here from google, you will want to follow this: urllib3/urllib3#9 (comment)

[petri]

This should be reopened - it's not out of scope, and it's quite necessary when working with hosts inside a VPN.

Depending on specifics of the address spaces of the current & remote network, one often has to send requests from the IP bound to the interface set up by the VPN. Or make changes to the routing table. This applies at least to using NetExtender on a Mac.

Many networking programs such as ssh and curl support setting the interface to use / binding to IP of an interface.

[Lukasa]

@petri Your assertion that this is not out of scope is incorrect, given that the scope of the Requests project is defined entirely by Kenneth. =)

Specifically, we view it as being out of scope for the core requests project. However, you can still do it, and it's not even very hard. This discussion covers how to do it.

[petri]

Nice - I was not aware of that kind of mechanism in requests. Thanks!

[danlesko]

import socket
import urllib2
true_socket = socket.socket

def make_bound_socket(source_ip):
    def bound_socket(*a, **k):
        sock = true_socket(*a, **k)
        sock.bind((source_ip, 0))
        return sock
    return bound_socket

socket.socket = make_bound_socket('<some source ip>')
print urllib2.urlopen('http://httpbin.org/ip').read()

socket.socket = make_bound_socket('<some other source ip>')
print urllib2.urlopen('http://httpbin.org/ip').read()

If traffic going out from the two source ip numbers goes out through two different public ip numbers one should see these different public numbers in the origin field of the response.

Just wanted you to know I've been scouring the internet for forever and this was the only solution that worked for me.

[sigmavirus24]

https://toolbelt.readthedocs.io/en/latest/adapters.html#sourceaddressadapter exists and is far less of a hack