warc: add Network.resourceType (https://chromedevtools.github.io/devt…

[ikreymer]

Add resourcesType value from https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-ResourceType as WARC-Resource-Type header, fixes #451

[tw4l]

Tested, looks good

[ikreymer]

Only question for me is if: https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-ResourceType is standard enough.

There is the extension-focused: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/ResourceType which is slightly more standardized (still differences between Chrome and Firefox) but its not a one-to-one mapping with the CDP resourceType that we have access to

[tw4l]

Only question for me is if: https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-ResourceType is standard enough.

There is the extension-focused: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/ResourceType which is slightly more standardized (still differences between Chrome and Firefox) but its not a one-to-one mapping with the CDP resourceType that we have access to

It's a good question, but if the CDP resourceType is what we have access to and we have a reference we can point to, I think that's probably sufficient at this stage, especially since we are only crawling using Chromium-based browsers. Unless there's a very clear standard to map to cleanly, which I don't know that there is in this case.

[ikreymer]

This is probably fine for now, but also opened an issue in WARC spec about possibly standardizing how this is approached. iipc/warc-specifications#96

[ikreymer]

Per discussion, it appears that Puppeteer / Playwright both use resourceType as all lowercase. To match their convention, also setting WARC-Resource-Type header to be all lowercase and updating the pageinfo record to store the lowercase version as well.