Possibility to extract other metadata than Persons, Organizations or Locations

[ali-akdag]

Hello,

I'm orientating on the feasibility of leveraging this product with other types of datasets. These PDF datasets are written based on a certain protocol and should contain the metadata we are looking for consistently, such as "locationcode", "city" and "documenttype".
I was wondering, instead of using the NLP for Persons, Organizations or Locations, would we be able to make changes to support this? This should also reflect back on the frontend in terms of filter and perhaps even columns in the searchpage/results.

[pirhoo]

Hi @ali-akdag, in fact this is on our roadmap for next few months!

We plan to add much more entities, both using NLP and less-heavy text processing method like Regex or document's metadata. This is a big move in terms of data model so we are planning accordingly. More to come soon :)

[ali-akdag]

This is great news! I believe this would make the application available for more generic purposes.

We currently have a similar application in use with 1M+ PDF files which we developed ourselves, using a custom frontend, Tika and Elasticsearch. It certainly is not as appealing as this product and lacks features such as OAuth and a desktop application.

Offtopic questions/remarks (sorry):

• With the PDF preview we use in our own application, the viewer (PDF.js) would jump to the searched word and highlight it immediately. Is this also on the roadmap or an idea?
• Is there a reason why an outdated version of Elasticsearch is used? Any upgrades planned?
• Any support planned for integration with sources such as Azure Storage Accounts? As I currently understand the documentation, you need to place new files in the batch directory which acts as a queue.

[pirhoo]

Excellent questions!

• We used to use PDF.js as well but we run into many troubles with big PDFs (they are many in our project) so we decided to fallback to the build-in web browser PDF. We have big plans to replace the text extraction pipeline to store more structured data so we could present PDF content in a more formatted ways ;
• We are stuck to an old version of ElasticSearch because their recent version cannot be use in "embedded" mode (sharing the same JVM as Datashare) which is a key feature for the portability of Datashare in local mode. Unfortunately if we don't find a solution we will have to fall back to a more tradition running mode (two JVMs both for Datashare and ES) which will have significant impacts for low-memory configurations.
• No plan to support Azure Storage Accounts or other queue type yet. The only change on that front is the current implementation of AMQP to handle asynchronous tasks.