Ensure licenses get inherited from parent targets that are not host-level URLs

[anjackson]

We have an example target which is a 'child' of another target that is open access:

https://twitter.com/rpoonline/
https://twitter.com/rpoonline/status/1544706478359068675/

This is showing up in this Collections page as 'available only in Reading Rooms'.

At the time of writing, we've not actually crawled the child URL, but I don't think this is the cause of the issue.

The problem may instead lie with this license logic:

python-w3act/w3act/dbc/generate/collections_solr.py

Lines 39 to 45 in 5975dd5

	# Determine license status:
	licenses = []
	if target.get('isOA', False):
	licenses = target.get("license_ids",[])
	# Use a special value to indicate an inherited license:
	if len(licenses) == 0:
	licenses = ['1000']

...generated at...

python-w3act/w3act/dbc/client.py

Line 325 in 5975dd5

targets[tid]['isOA'] = check_oa_status(targets[tid])

...but this gets populated at...

python-w3act/w3act/dbc/client.py

Lines 330 to 341 in 5975dd5

	# Second pass to add inherited statuses:
	# FIXME Both should be inherited from all higher-level Targets. This version only inherits from hosts.
	for tid in targets:
	for url in targets[tid].get('urls',[]):
	parsed_uri = urlparse(url)
	base = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
	if base in oa_urls and not targets[tid]['isOA']:
	targets[tid]['isOA'] = True
	targets[tid]['inheritsOA'] = True
	if base in npld_urls and not targets[tid]['isNPLD']:
	targets[tid]['isNPLD'] = True
	targets[tid]['inheritsNPLD'] = True

NOTE the FIXME that this version only inherits from host-level records. So this looks like an old outstanding issue that needs to be resolved.

[anjackson]

Hah, no, this isn't what's going on. Complete rewrite of issue needed!