Anaphora resolution

[schoen]

I recently learned about Winograd Schemas

https://en.wikipedia.org/wiki/Winograd_Schema_Challenge

which are an approach to formalizing the testing of anaphora resolution (figuring out what a pronoun refers to when this can only be done from context and knowledge about the world). For example, in the pair of sentences

The doctor examined the patient because she felt ill.

The doctor examined the patient because she was familiar with endocrine disorders.

the word she in the first sentence refers to the patient, but in the second sentence to the doctor, because feeling ill is a reason for someone to be examined by a doctor, while medical expertise is a reason for someone to examine a patient. Or in a somewhat similar vein

Mahayana Buddhist monks may not eat carne asada burritos because they are not vegetarian.

Mahayana Buddhist monks may not eat tacos al pastor because they are vegetarian.

(In one case, they [are/are not] vegetarian refers to people, and in the other case to food items. For the Winograd schema case we would probably just say "Mahayana Buddhist monks may not eat carne asada burritos because they [are/are not] vegetarian", where then the word they will refer either to people or to food items.)

This is a remarkably difficult natural language processing problem because particular instances of it can test arbitrary knowledge about the world and the ability to make likely inferences about relationships, institutions, abilities, and motivations.

In Winograd schemas, a single change in a sentence flips the most natural resolution of a pronoun to refer to a different entity, and the question is to figure out which entity is referred to by each version of the sentence.

The Wikipedia article shows that there are several attempts to test how well different AIs and natural language processing systems can solve Winograd schema problems. Are any tests of this kind already included in any of the reading comprehension tasks? If not, they probably should be!

[pde]

It looks like anaphora resolution would be a subproblem of the reading_comprehension problem, and each Winograd Schema problem (a specific language and dataset) would be added as a metric on that problem.

[pde]

Go ahead and send a pull request with some data :)

[schoen]

Here is an example from the biennial challenge

http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml

I'm not sure where to find the results of this challenge, but it is being run periodically.

[schoen]

The organization of public competitions in this area is at

http://commonsensereasoning.org/disambiguation.html
http://commonsensereasoning.org/winograd.html

and I'm not sure where the precise results have ended up.