How to: news metadata -> full text -> pipeline

[kmcdono2]

Ticket to strategize getting a sample of full text articles into the toponym resolution pipeline.

[kallewesterling]

Just to start a thread here for myself: This pull request is where @thobson88 has created a workflow for creating fulltext for a given set of items, based on locally downloaded zip files. (Not the best for the pipeline outlined here).

Meanwhile, I'm working on ingesting the fulltext into a separate table. (And then @thobson88's method will have to be used inside a VM for BNA access somehow!)

[kmcdono2]

@kallewesterling let us know how we can help test this method, when it's useful!

[kallewesterling]

Thanks, definitely @kmcdono2! We're almost there with the fulltext table as well, which will be available outside of the BNA!

[kmcdono2]

@kallewesterling @lukehare can we add some information here about how this process will work with the new API for toponym resolution?

[kallewesterling]

Yeah that's a great point. We need to keep this in mind. I assume we could just tag it on outside the db code base (separation of concern, makes the most sense to me) but can also think of ways to build it into the db code (which would be easier for the end user)... food for thought. Let's chat more!

[ruthahnert]

Would love to chat more about framing narratives

[kmcdono2]

That would be great @ruthahnert! will follow up on slack, but you can see here and in other tickets how we are trying to prep for understanding this as part of working with newspaper full text content