OVERALL_DISCARD_XPATH not discarding in some cases

[felipehertzer]

Hi @adbar,

I encountered this problem when trying to scrape this site.

The code is bringing the sidebar with the body, the sidebar div class is 'l-sidebar'.

So in my tests I arrived at this function:

tree = prune_unwanted_nodes(tree, OVERALL_DISCARD_XPATH, with_backup=True)

If I set with_backup to False it works.

It may be related to this TODO:

# todo: adjust for recall and precision settings
if new_len > old_len/7:
    return tree

example 2

[adbar]

Hi @felipehertzer, as with the other heuristics it's tricky, the number 7 is chosen arbitrarily, let's see if we can find a better way or another threshold.