HTTPS: Server Name Indication (SNI) support?

[michieldetailleur]

If I'm not mistaken linkchecker (or the http library that linkchecker uses) does not support Server Name Indication ("vhost support for https") at the moment?

It fails with the following error:

Error: SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Could support for SNI be added? (or be listed as a "known issue" ;) )

Thank you,
Michiel

[wummel]

We believe that the issue you reported is fixed in the source repository of linkchecker which can be found under:
https://github.com/wummel/linkchecker

Changelog entry:

• checking: Use the Python requests module for HTTP and HTTPS requests.
  Closes: GH bug HTTPS: Server Name Indication (SNI) support? #393

Thank you for reporting the issue. It is now marked as fixed. If you believe that the issue is not fixed appropriately just add a comment to this issue.

[wummel]

A new version 9.0 of linkchecker has been released on 3.3.2014.
Therefore this bug will be closed. If you think this issue is not solved, please open a new issue.

[aheissenberger]

I still get this error on our SNI enabled Webhost:

linkchecker --no-status --no-warnings https://www.conda.eu/
INFO 2015-02-24 08:25:20,518 MainThread Checking intern URLs only; use --check-extern to check extern URLs.
LinkChecker 9.3              Copyright (C) 2000-2014 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' within this
distribution.
Get the newest version at http://wummel.github.io/linkchecker/
Write comments and bugs to https://github.com/wummel/linkchecker/issues
Support this project at http://wummel.github.io/linkchecker/donations.html

Start checking at 2015-02-24 08:25:20+000

URL        `https://www.conda.eu/'
Real URL   https://www.conda.eu/
Check time 0.197 seconds
Result     Error: SSLError: hostname 'www.conda.eu' doesn't match either of 'www.conda.de', 'conda.de'

Statistics:
Downloaded: 0B.
Content types: 0 image, 0 text, 0 video, 0 audio, 0 application, 0 mail and 1 other.
URL lengths: min=21, max=21, avg=21.

That's it. 1 link in 1 URL checked. 0 warnings found. 1 error found.
Stopped checking at 2015-02-24 08:25:20+000 (0.25 seconds)

There is no error if I use curl:

curl -I -v https://www.conda.eu
* Rebuilt URL to: https://www.conda.eu/
* Hostname was NOT found in DNS cache
*   Trying 108.61.171.240...
* Connected to www.conda.eu (108.61.171.240) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*    subject: serialNumber=S142h9sruA-c3P68Jtr6p76aFlsHvTxf; OU=GT81966185; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=*.conda.eu
*    start date: 2014-10-26 10:34:13 GMT
*    expire date: 2015-10-29 04:11:16 GMT
*    subjectAltName: www.conda.eu matched
*    issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*    SSL certificate verify ok.
> HEAD / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: www.conda.eu
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
Server: nginx
< Date: Tue, 24 Feb 2015 08:34:16 GMT
Date: Tue, 24 Feb 2015 08:34:16 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Powered-By: PHP/5.6.5-1+deb.sury.org~trusty+1
X-Powered-By: PHP/5.6.5-1+deb.sury.org~trusty+1
< Last-Modified: Tue, 24 Feb 2015 07:49:56 GMT
Last-Modified: Tue, 24 Feb 2015 07:49:56 GMT
< Expires: Tue, 24 Feb 2015 08:49:56 GMT
Expires: Tue, 24 Feb 2015 08:49:56 GMT
< Pragma: public
Pragma: public
< Cache-Control: max-age=940, public
Cache-Control: max-age=940, public
< X-Pingback: https://www.conda.eu/xmlrpc.php
X-Pingback: https://www.conda.eu/xmlrpc.php

< 
* Connection #0 to host www.conda.eu left intact

The System is: Ubuntu 14.04.2 LTS

[VladimirAlexiev]

I get a similar error: sgvps.net is our hosting provider.

linkchecker https://ontotext.com
URL        `https://ontotext.com'
Real URL   https://ontotext.com
Check time 1.033 seconds
Result     Error: SSLError: hostname 'ontotext.com' doesn't match either of '*.sgvps.net', 'sgvps.net'

curl -I -v https://ontotext.com
* Server certificate:
*  subject: C=BG; ST=Sofia; L=Sofia; O=Ontotext AD; OU=Infra; CN=*.ontotext.com
*  start date: Jan 17 00:00:00 2017 GMT
*  expire date: Feb 16 23:59:59 2018 GMT
*  subjectAltName: host "www.ontotext.com" matched cert's "*.ontotext.com"
*  issuer: C=US; O=GeoTrust Inc.; CN=GeoTrust SSL CA - G3
*  SSL certificate verify ok.

[VladimirAlexiev]

@wummel httpie/cli#262 describes a similar error and talks about

pip install --upgrade requests[security]

I'm not sure where to look for this module, but I looked in c:\Program Files (x86)\LinkChecker\library.zip\requests\ and I don't see security.pyo.
So maybe this is not really fixed?

[VladimirAlexiev]

I'm wrong. I ran the above pip install and checked c:\users\vladimir.alexiev\appdata\local\programs\python\python35\lib\site-packages\requests and the only 2 extra files compared to the linkchecker dir above are help.py packages.py.

[VladimirAlexiev]

Same as linkchecker/linkchecker#66, #715?

[VladimirAlexiev]

setting sslverify=0 works, so that's an acceptable workaround.