Connection reuse in proxy mode

[ecktom]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Network project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

When using Oathkeeper in proxy mode and putting some higher RPS on it we noticed that there is something fishy w.r.t. to connection reuse. We saw plenty of new connections getting established, leading to higher load on CoreDNS etc.

On a fast glance, we think it could be related to https://github.com/ory/oathkeeper/blob/master/proxy/proxy.go#L74. Go's DefaultTransport has an insane default setting for idle connections. While it allows up to 100 idle connections, the number is limited per host to 2:

var DefaultTransport RoundTripper = &Transport{
	...
	MaxIdleConns:          100,
	...
}
// DefaultMaxIdleConnsPerHost is the default value of Transport's
// MaxIdleConnsPerHost.
const DefaultMaxIdleConnsPerHost = 2

I would propose using something like

t := http.DefaultTransport.(*http.Transport).Clone()
t.MaxIdleConns = 100
t.MaxIdleConnsPerHost = 100
	
res, err := t.RoundTrip(r)

Reproducing the bug

See Description

Relevant log output

No response

Relevant configuration

No response

Version

0.40.0

On which operating system are you observing this issue?

None

In which environment are you deploying?

None

Additional Context

No response