You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm running a test project on localhost:8000 and when I access it over https, it fails (which is expected)
Get "https://localhost:8000/": tls: first record does not look like a TLS handshake
The above is correctly caught in OnError. However, when I set ignoreRobots to false, then it tries to fetch the robots.txt and the below failure
Get "https://localhost:8000/robots.txt": tls: first record does not look like a TLS handshake
Is not propogated to OnError - as it is really not originating from the request that I had started, but colly tries to first fetch the robots which fails.. Could this also be propogated either to OnError or can be caught with a known Error Code from Colly such as
ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
ErrRobotsTxtFetchFailed = errors.New("Unable to fetch robots.txt") // New Error Code
The text was updated successfully, but these errors were encountered:
I'm running a test project on localhost:8000 and when I access it over https, it fails (which is expected)
Get "https://localhost:8000/": tls: first record does not look like a TLS handshake
The above is correctly caught in
OnError
. However, when I setignoreRobots
tofalse
, then it tries to fetch therobots.txt
and the below failureGet "https://localhost:8000/robots.txt": tls: first record does not look like a TLS handshake
Is not propogated to
OnError
- as it is really not originating from the request that I had started, but colly tries to first fetch the robots which fails.. Could this also be propogated either toOnError
or can be caught with a known Error Code from Colly such asThe text was updated successfully, but these errors were encountered: