Skip to content

A symptom tracking app for common respiratory flu symptoms, especially COVID-19

License

Notifications You must be signed in to change notification settings

markusthoemmes/hatschi

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hatschi

Hatschi (cutesy german way of saying "bless you") is an application that automatically tracks respiratory symptoms of the cold, the flu or COVID-19 in particular. It automatically assesses these symptoms through usage of a microphone and gives the user a statistical overview of the amount of symptoms shown throughout the day. That data is furthermore used to give health recommendations to the user, i.e. to stay at home or to get tested for COVID-19.

Try it out here.

Note: The application doesn't actually detect coughs yet (see explanation below). You can say "three" to make it count "coughs" though.

Note: The application doesn't send data anywhere. The code runs locally in your browser and historic data is stored in your browser using IndexedDB. Nobody else will ever get access to the audio stream analyzed here neither see historic data of your health status.

The idea

We cough, sneeze and sniff subconsciously throughout the day. Sometimes we don't even know if we're healthy or sick. We might feel healthy but actually we're a walking germ-bomb. This application puts an end to that and tracks your respiratory symptoms throughout the entire day. It's meant to run in the background and go unnoticed. Based on the data it gathers, it's able to tell if you're healthy (having a very low symptom count), kinda unhealthy or if you should go see a doctor immediately. This is especially important in the impeding COVID-19 outbreak but not necessarily limited to it at all. It could be used in a day-to-day office setting as well, where it would prevent workers from being in the office if they are infectuous.

Moreover, we can also qualitatively analyze the type of coughs. A dry-cough is characteristic for COVID-19 for example. We can use our detection to either urge the user to go to see a doctor or to calm them down by stating that the cough pattern is not indicative of COVID-19.

Detecting respiratory symptoms

There is quite a bit of scientific research in this area.

Based on this research it should be possible to accurately detect different respiratory symptoms. Given the shortage of time (the hackathon had 48 hours of code-time) we went for a Tensorflow.js based approach to make it runnable in the browser and thus available on mobile and laptop/desktop pcs.

We collected sample data from Google Research and trained a model to detect coughs. Unfortunately this is where the time constraint bit us and we had to leave it up to be implemented post-hackathon if needed. We have a rough idea on how to implement a model now, as can be seen by the model directory in this repository. The model in there is a very crudely trained model of the spoken words model used in the application (see below) though. Training a proper cough model would take longer than the deadline permits at this point.

Currently, the implementation uses a Tensorflow.js sample model, that is able to detect spoken words. The word "three" is used as a synonym of a cough in the current implementation.

The hackathon process

We started at 0, with a team of 8. We divided the team into a visualization and a model team. The former would design and implement the application, get the statistics and visualize it for the the user. It also implemented any decision making logic. The model team was responsible for collecting training data and producing a model to be used in TensorflowJS.

Working in parallel has greatly improved the throughput of our work and we've been able to make great progress on both fronts, as can be seen in the example above. We underestimated the interface between a generic Tensorflow model and TensorflowJS though which at the end was a dealbreaker for not being able to import the trained model into the application.

Future work

Implementing the cough detection

As mentioned above, we didn't succeed in training the cough model though in theory, given more time, that should absolutely be possible. Relevant resources can be found here and here. These examples need to be adjusted to not work on the list of words they are based on now, but to be trained to detect "cough". This repository does not contain the cough sample set we created due to potential copyright infringement. This repository started a collection on covid/non-covid coughs to train models for example.

Implementing more than just cough detection

The model could not only detect cough, but also which kind of cough (i.e. dry-cough) or other respiratory symptoms like sneezes or rattling. All of that data can greatly improve the recommendation engine to give deeper insight and more nuanced guidance.

Improving the recommendation engine

The recommendation engine is currently very crude and only take a few values into account. You might find the recommendations it gives very strange sometimes, especially when using a randomized data set. This should be improved upon using more insight from actual medical professionals and respective statistical handling of the data at hand.

Collect anonymized data centrally

As a lot of symptom-gathering apps are built these days, this application could of course be used as well to feed a centralized data store to determine hot-spots of flu-like outbreaks and even COVID-19 spread. It could even help to narrow down the number of unreported cases and give a clearer picture on how many people are actually likely infected.


This app has been created as part of the "WirVsVirus" Hackathon held by the federal government of Germany to fight the corona virus disease.

About

A symptom tracking app for common respiratory flu symptoms, especially COVID-19

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published