You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using Request Context to store information about the parsed body on various c.OnHTML callbacks..
So what happens is, if I use the e.Request.Visit() for following up on hrefs, then the request context is also being passed. I wanted to avoid this. So instead of using e.Request.Visit() I used c.Visit() directly. This made sure that I got new context for each request.
However, I would like to use the MaxDepth option as well. But that only works if I use the e.Request.Visit().
It would work for me to use the e.Request.Visit() but give new context for each request. This is currently not possible. Is that correct?
If yes, this feature request would be great to have as a configuration option - to determine if the request context has to be passed along or not..
For now, I have manually made the change for local purposes..
index 6beef834..524bb77c 100644
--- a/vendor/github.com/gocolly/colly/v2/request.go+++ b/vendor/github.com/gocolly/colly/v2/request.go@@ -117,7 +117,7 @@ func (r *Request) AbsoluteURL(u string) string {
// request and preserves the Context of the previous request.
// Visit also calls the previously provided callbacks
func (r *Request) Visit(URL string) error {
- return r.collector.scrape(r.AbsoluteURL(URL), "GET", r.Depth+1, nil, r.Ctx, nil, true)+ return r.collector.scrape(r.AbsoluteURL(URL), "GET", r.Depth+1, nil, nil, nil, true)
}
// HasVisited checks if the provided URL has been visited```
The text was updated successfully, but these errors were encountered:
sundarv85
changed the title
Option not to oass Request Context to the Next Request
Option not to pass Request Context to the Next Request
Nov 30, 2022
I'm using Request Context to store information about the parsed body on various
c.OnHTML
callbacks..So what happens is, if I use the
e.Request.Visit()
for following up on hrefs, then the request context is also being passed. I wanted to avoid this. So instead of usinge.Request.Visit()
I usedc.Visit()
directly. This made sure that I got new context for each request.However, I would like to use the
MaxDepth
option as well. But that only works if I use thee.Request.Visit()
.It would work for me to use the e.Request.Visit() but give new context for each request. This is currently not possible. Is that correct?
If yes, this feature request would be great to have as a configuration option - to determine if the request context has to be passed along or not..
For now, I have manually made the change for local purposes..
The text was updated successfully, but these errors were encountered: