Make PageCrawler a "howto" instead of built-in feature?

[chillu]

Looking at PageCrawler, that's a non-trivial amount of code (50NCLOC), so somewhat valuable for devs to avoid writing those particular XPATH incantations. But I don't think we need to bake the actual processing logic triggering this utility into the module. At the moment, DataObjectDocument has a page_content_field which gets populated conditionally on shouldCrawlPageContent().

My preference would be that we tell people how to add a getter to their Page class (or anything else that could be "crawled") which can instanciate a PageCrawler, and explicitly assign it to a field in the search schema of their choosing. The advantage of this approach is that the crawler can be adapted or replaced, e.g. filtering out certain tags, extracting image titles, etc. It's also closer to what we already recommend for text extraction on File records.

[chillu]

If we wanted to avoid hardcoding this logic in DataObjectDocument, we could also generalise this as a "resolver" for a field.

interface DataObjectFieldResolver
{
  public function resolve(DataObject $obj): mixed;
}

Then we could have a PageCrawlerFieldResolver than can be applied through YAML configuration, rather than writing a new getter in PHP, and it can be used on multiple types (not only pages can have URLs). A resolver pattern could also be handy for more complex lookups, e.g. transforming a "many many tags" relationship into an array of tag titles