Option to include all tags and attrs in LinkExtractor with specified exclusions

[User087]

Summary

Add the option to the LinkExtractor class to consider all tags and attributes (e.g. if you pass None then consider all tags/attributes), and deny_tags and deny_attrs arguments or similar so you can additionally consider all tags and attributes with the exception of those explicitly passed.

Motivation

It allows adopting a strategy of extracting all links by default and then specifically excluding the tags and attributes you don't want considered. Currently, it seems the user has to figure out all the specific tags and attributes where they're desired links appear and explicitly pass them to tags and attrs to have them considered.

Describe alternatives you've considered

For including all tags, you could use the Selector class instead of LinkExtractor and select all e.g. href attributes regardless of which tag they appear in, e.g. response.xpath('//@href'). Using Selector results in losing the various convenient arguments in LinkExtractor and requires manually processing them with regex etc instead, and it requires manually converting relative links into absolute links when you want to use regexes that match the entire URL whereas LinkExtractor already handles that automatically.

Additional context

Any additional information about the feature request here.

[PJ1256]

I would like to try and work on this if that's ok.

[PredictiveManish]

I am trying to solve this issue.

[parthvichare]

Its great challenging problem, I love to work on it

[Noman654]

@Laerte, I'd like to tackle this issue! As it's my first contribution to the project, any pointers to get me started would be much appreciated.

[Laerte]

Hi @Noman654 seems that we already have a open PR for it:

• Issue #6321: Link extractor all tags and attributes option #6327

[Gallaecio]

#6327 is about the first part, making None include all.

The deny part could be implemented separately by someone else, I think. There could be conflicts, but they should be easy to resolve. I do think a boolean reverse_filter parameter would be better than 2 new parameters to implement that behavior, though.

[ShubhGohil]

Hi, I had a question and suggestion about this enhancement.

Question:
Wouldn't making the LinkExtractor extract all the tags and attributes as its default behaviour, and at the same time having the options to select and omit specific tags and attributes make it a bit complex?

Suggestion:
Rather can we have separate classes for specific jobs.

1. One class extracts all the links and attributes as its default behavior and gives the option to filter out the tags and attributes not required.
2. Other class which accepts a list of tags and attributes to be extracted.

Please let me know your thoughts on this. Or the thought process behind having all the options in a single class.

@Gallaecio , the idea to keep a boolean reverse_filter would increase the conditional statements in the class but can be implemented.

[Gallaecio]

I think the argument for complexity has merit, but I don’t think a class parallel to LinkExtractor would be the way to keep complexity manageable. Maybe we could create a LinkExtractorFilter class, allow passing as instance of that as the value of the filter parameter, and move all filtering-related complexity to that class.

No strong opinion, though. As long as the user API is simple enough, I am open to any proposal.

[ShubhGohil]

@Gallaecio Could you please expand on how the LinkExtractorFilter class would work?

If we do not wish to create two separate classes then we would have to include two extra parameters deny_tags and deny_attrs as mentioned originally in the issue.

We have different conditions while filtering,

1. if we want to include some tags and include some attributes.

The above condition is the current default behaviour

2. if we want to extract all the tags and all the attributes
3. if we want to exclude some tags and exclude some attribute.

The above two conditions can be infused by introducing a boolean filter parameter. Condition 2 would become the default behaviour in this case

4. if we want to include some tags and exclude some attributes from within those elements
5. if we want to exclude some tags and include some attributes from the elements extracted

The above two conditions would need the use of two different parameters deny_tags and deny_attrs

Just trying to break down the problem and make it easier to navigate.

[Gallaecio]

Could you please expand on how the LinkExtractorFilter class would work?

I’m not sure. I just think there must be an alternative to having 2 separate link extractor classes based on whether the filtering is inclusive or exclusive.

if we want to include some tags and exclude some attributes from within those elements
if we want to exclude some tags and include some attributes from the elements extracted

That makes sense. But if we want maximum flexibility, we also need the ability to allow/deny specific combinations of tags / attributes. And it complicates things, with no real-life use case to justify that complexity.

But I feel like I am rambling now. I have no concrete idea of how to best address the issue, to be honest. I’m more than happy to review any PR for it, though.