Healthsea: a pipeline for extracting potential health benefits in text

[thomashacker]

Healthsea is an end-to-end spaCy pipeline for analyzing supplement reviews to extract potential health benefits! It uses spaCy's built-in NER and Text Classification capabilities together with custom components for Clause Segmentation and Entity Blinding ✨

📖 Read more

• Blog post: The journey of building the Healthsea pipeline
• Healthsea demo: Demo for showcasing Healthsea's results on real data
• Healthsea pipeline: Demo for exploring the individual steps of the pipeline.
• Github: A GitHub repo containing a spaCy project for you to clone and rebuild Healthsea yourself

[ytsaig]

Very cool project and great writeup. Since you asked to try and break it :) here's one example I came across:

doc = nlp("This did not help my joint pain.")

Yields the expected (negative) result:

'cats': {'POSITIVE': 0.04110577329993248, 'NEUTRAL': 0.9418349266052246, 'NEGATIVE': 0.007117656525224447, 'ANAMNESIS': 0.009941641241312027}

But changing the wording to

doc = nlp("This has not helped my joint pain.")

Flips the class probabilities:

'cats': {'POSITIVE': 0.9939416646957397, 'NEUTRAL': 0.005660131108015776, 'NEGATIVE': 0.0001686908508418128, 'ANAMNESIS': 0.00022952131985221058}

[thomashacker]

Awesome! That's a good example. 🎉
The first prediction returns NEUTRAL because we have no effect on the entity joint pain, it would've been NEGATIVE if the joint pain got worse.

It's interesting that the prediction completely flipped after changing the wording, which makes me think if normalizing text might improve performance or if more training data could do the trick.

Thanks for breaking my model 😄

[enodr]

Hi,

In the blog post you write:
To find aspects of health, we need to define them. The first concepts that come to mind are diseases and symptoms, which you generally want to improve by ”decreasing” them. The ICD-11 (International Classification of Diseases) is a database that holds a wide variety of health information about diseases and symptoms. It provides an API that you can use to retrieve the data.

But I don't see anywhere in the code where the ICD-11 database is used as a Knowledge Base. Does the published model relies only on annotations to define the CONDITIONS ?

[thomashacker]

Hey, thanks for the question.

Yes, the current model only relies on the NER annotations. However, we used a small extract from the ICD-11 to help annotating, but since we also want to detect more vague conditions that fall out of the ICD-11 we couldn't fully rely on it.

Nevertheless, adding the Knowledge Base to the model would be a great addition 🎉