Review Policies

Review Policies allow requesters to setup decision rules by which assignments are accepted and rejected. They come in two flavors: HIT-level review policies and Assignment-level review policies. Currently, the only HIT-level review policy is based on inter-worker consensus and has obvious applications to human coding of data. Currently, the only Assignment-level review policy is based on "known answers," which allows the requester to accept or reject assignments based on answers to any or all questions for a HIT.

The MTurk documentation includes some example use cases for review policies.

HIT-level Review Policies

A HIT-level review policy can be attached to a HIT by specifying hit.review.policy in CreateHIT. This argument must be a character string containing a valid HITReviewPolicy data structure.

Below are some code examples demonstrating how to use HIT-level ReviewPolicies.

Conditionally extend HIT based on HIT Agreement Score

lista <- list(QuestionIds = c("Question1","Question2","Question5"),
              QuestionAgreementThreshold = 49, # at least 50 percent agreement
              ExtendIfHITAgreementScoreIsLessThan = 50,
              ExtendMinimumTimeInSeconds = 3600,
              ExtendMaximumAssignments = 2,
              DisregardAssignmentIfRejected = TRUE)
policya <- do.call(GenerateHITReviewPolicy, lista)

hit1 <- 
CreateHIT(title = "Survey",
          description = "5 question survey",
          reward = ".10",
          expiration = seconds(days = 4),
          duration = seconds(hours = 1),
          keywords = "survey, questionnaire",
          hit.review.policy = policya,
          question = GenerateExternalQuestion("http://www.example.com/","400"))

Conditionally approve and reject based on Worker Agreement Score

listb <- list(QuestionIds = c("Question1","Question2","Question5"),
              QuestionAgreementThreshold = 65, # at least two of three 'correct' answers
              ApproveIfWorkerAgreementScoreIsAtLeast = 65,
              RejectIfWorkerAgreementScoreIsLessThan = 34,
              DisregardAssignmentIfRejected = TRUE)
policyb <- do.call(GenerateHITReviewPolicy, listb)

hit1 <- 
CreateHIT(title = "Survey",
          description = "5 question survey",
          reward = ".10",
          expiration = seconds(days = 4),
          duration = seconds(hours = 1),
          keywords = "survey, questionnaire",
          hit.review.policy = policyb,
          question = GenerateExternalQuestion("http://www.example.com/","400"))

See the MTurk Documentation for further details.

Assignment-level Review Policies

An Assignment-level review policy can be attached to a HIT by specifying assignment.review.policy in CreateHIT. This argument must be a character string containing a valid HITReviewPolicy data structure, including an AnswerKey, which indicates correct or "known" answers to specified questions.

Below are three example Assignment-level review policies, which can all be attached to a HIT using the assignment.review.policy argument to CreateHIT.

Conditional approval of assignments based on percent of correct answers

lista <- list(AnswerKey = list("QuestionId1" = "B",
                               "QuestionId2" = "A"),
              ApproveIfKnownAnswerScoreIsAtLeast = 99)
policya <- do.call(GenerateAssignmentReviewPolicy, lista)

Conditional rejection of assignments based on answer score

listb <- list(AnswerKey = list("QuestionId1" = "B",
                               "QuestionId2" = "A"),
              RejectIfKnownAnswerScoreIsLessThan = 1)
policyb <- do.call(GenerateAssignmentReviewPolicy, listb)

Conditionally extend HIT based on answer score

listc <- list(AnswerKey = list("QuestionId1" = "B"),
              ExtendIfKnownAnswerScoreIsLessThan = 100,
              ExtendMaximumAssignments = 2, # maximum value is 25
              ExtendMinimumTimeInSeconds = seconds(hours = 1))
policyc <- do.call(GenerateAssignmentReviewPolicy, listc)

Note: Assignment-level review policies can only be attached QuestionForm and ExternalQuestion HITs (and thus not those created through the Requester User Interface).

See the MTurk Documentation for further details.