Feature/adding relevance scores

[tgalery]

This PR adds a feature to dbpedia-spotlight, namely, weights associated with the annotations extracted.To do so, all you need to do is add a line containing relevance_scoring=default to your model.properties file in the model folder. If that line is not present, spotlight behaves as usual.

We use:

• the context vector overlap
• number a times a topic is spotted in the text
• the overlap among context words for the topics. For example "Microsoft" being a context words for the topics.
• We normalize the output score using min-max normalization (0-1) ( this step could be improved)

Given some toy data that we had manually annotated, we realized that this method gives results close to human judgements than those given by other topic extractors such as Zemanta or Alchemy. If you think this is a good idea, we could manually annotate some establish dataset, like the milne witten, and write a paper as an attempt to reproduce the results in a more formal way.

[jodaiber]

This looks good, any open issues?

[tgalery]

Hi @jodaiber no open issues that I know of. I could rebase on master, but I wonder whether the missing config would break the branch, as in the other pr.
Would be good for other people to test this to see if it's working as expected.

[pablomendes]

@tgalery, this is great! I have wanted to add relevance for a while now,
but it was always trumped by other more serious issues. Never got to it!

Luis Marujo had a paper and shared a number of datasets in his LREC2012
paper that we could try to use for evaluation if their definition of
relevancy is at all related to yours.

I would love to help with the paper however I can.
On May 13, 2014 10:08 AM, "tgalery" notifications@github.com wrote:

Hi @jodaiber https://github.com/jodaiber no open issues that I know of.
I could rebase on master, but I wonder whether the missing config would
break the branch, as in the other pr.
Would be good for other people to test this to see if it's working as
expected.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/300#issuecomment-42983399
.

[tgalery]

Thanks @pablomendes ! Glad to have such nice feedback. I'm gonna take a look at the Majuro paper when I have a chance and see if I can re-use his dataset. I will keep you posted on how things develop here (I guess I can find your email in google, yeah?). Meanwhile if you or @jodaiber want to test the branch more and give feedback on the scores you get back, feel free to do so.

[tgalery]

hi @pablomendes I was taking a look at the Majuro dataset and apparently it can be found here https://github.com/snkim/AutomaticKeyphraseExtraction . However, when I openned the data, it didn't seem to contain the relevance scores.

I just took a look at the Roder et al paper and it seems that they have some publically available datasets for keyword extraction and disambiguation. I was wondering if maybe using that in a mechanical turk interface to collect the data would be a good idea. Or else maybe contacting Majuro himself.