Exercise 2.1: Evaluation of Aggregation

[pintersebastian]

Hi! I have a question on how we should evaluate our aggregation results!
In the description it says:

Analyze the results of the aggregation, by randomly(!) picking some examples and conducting a meta-judgement of the raw-judgements + the quality of the aggregations based on the text contents of query and passage

I dont really understand much here...

Should we just compare our aggregation with the tf of the query terms in the respective document? That would only be

the quality of the aggregations based on the text contents of query and passage

What is meant with the first part of the description? Namely:

Analyze the results of the aggregation, by randomly(!) picking some examples and conducting a meta-judgement of the raw-judgements

Thanks in advance!

[ldosen00]

You are expected to randomly select a few of the examples, read the text content of the query and decide whether it is accurate(whether they reflect the true meaning of the text). That is meant by meta-judgment.
The intention behind this is to just make sure that the aggregation was done correctly and that there are no errors which would cause problems later in your solution.

[pintersebastian]

Alright, thanks a lot!