Dynamic Qualifications

Sometimes you may have multiple similar HITs that you would like to make available to workers but you would only like a given worker to complete one of those HITs (or some subset thereof). For example, perhaps two HITs represent different experimental conditions and you would like to randomly make only one of the HITs available to a worker. One standard approach to this problem is to create a Qualification (perhaps using a Qualification Test) and attach different QualificationRequirement values to each HIT based on that Qualification. Another approach, described here, involves creating a requestable Qualification with no test and then randomly assign values for the Qualification to each worker who requests the Qualification. This tutorial walks through how to dynamically, randomly assign Qualification values to workers whenever they request the Qualification, thus randomly making one of two HITs available to each worker.

Step 1. Create Qualification

The first step is to create a requestable QualificationType:

qual1 <- CreateQualificationType(name="Requestable, No-test Qualification",
           description="Just request this Qualification and it will be granted momentarily",
           status = "Active",
           keywords="requestable, no-test, quick, easy")

Step 2. Create HITs

Then, we need to create two HITs, where each HIT has a different QualificationRequirement value for the new Qualification we just created. We'll start by creating two QualificationRequirement objects:

score1 <- GenerateQualificationRequirement(qual1$QualificationTypeId, "==", "1", preview = TRUE)
score2 <- GenerateQualificationRequirement(qual1$QualificationTypeId, "==", "2", preview = TRUE)

Note the preview = TRUE option means that workers will not be able to view the contents of each HIT unless they have the Qualification at the specified value. This prevents cross-condition contamination.

Next we create two HITs (one using each of the above QualificationRequirements):

hit1 <- CreateHIT(  
  question = GenerateExternalQuestion("https://www.example.com/HIT1.html","400")$string,
  annotation = "Random Assignment Experiment",
  assignments = "50",
  title = "A survey about social issues in the United States",
  description = "A survey about social issues in the United States",
  reward = ".50",
  duration = seconds(hours=1),
  expiration = seconds(days=1),
  keywords = "survey, question, answers, research, politics, opinion",
  auto.approval.delay = seconds(days=1),
  qual.req = score1 # QualificationRequirement `score1`
)
hit2 <- CreateHIT(  
  question = GenerateExternalQuestion("https://www.example.com/HIT2.html","400")$string,
  annotation = "Random Assignment Experiment",
  assignments = "50",
  title = "A survey about social issues in the United States",
  description = "A survey about social issues in the United States",
  reward = ".50",
  duration = seconds(hours=1),
  expiration = seconds(days=1),
  keywords = "survey, question, answers, research, politics, opinion",
  auto.approval.delay = seconds(days=1),
  qual.req = score2 # QualificationRequirement `score2`
)

Step 3. Poll for Qualification Requests

With the two HITs created, they will now be visible to workers but workers will not be able to preview or accept either HIT because no workers are qualified to view or complete them. When a worker views the HITs, they'll see something like the following:

If a worker clicks the Why? link, they'll see a screen explaining why they cannot access the HIT:

And from there or the original HIT screen, they can request the qualification, which will lead them to a screen like this:

If they click Continue, they'll receive a confirmation message indicating that a QualificationRequest has been sent to you (the requester):

In MTurkR, we can retrieve this QualificationRequest using GetQualificationRequests:

(qrs <- GetQualificationRequests())
# 1 Requests Retrieved
#           QualificationRequestId            QualificationTypeId      SubjectId           SubmitTime Answer
# 2 32912ICPSFFYQNOSJDHCFQGVZNHLUH 3T04ZEB6XQ81KXFC1TYHYG6T9GXA58 A1RO9UJNWXMU65 2015-03-29T21:16:20Z   <NA>

The output shows the QualificationRequestId which can be used to grant or reject the QualificationRequest. Note that GetQualificationRequests uses SubjectId instead of WorkerId (for some inexplicable reason).

To manually grant the QualificationRequest, we simply pass the QualificationRequestId value to GrantQualification along with the specified Qualification value:

GrantQualification(qrs$QualificationRequestId[1], values = 1)

This is inconvenient to do manually, however. So instead we can use a loop to repeatedly retrieve all new QualificationRequests and then randomly assign a value to each of them, thus opening one (and only one) of the two HITs to each worker. Some key points:

1. We will make the loop conditional on the completion of all assignments for both HITs so that it exits once all assignments are completed.
2. We will use Sys.sleep to delay each iteration of the while loop. This way we poll for new requests periodically, thus reducing the number of iterations and number of API calls.

x <- 0
while(x < 100) {
    g <- GetQualificationRequests()
    if(nrow(g) > 0) {
        # randomly assign value
        s <- sample(1:2, nrow(g), replace = TRUE)
        # grant requests
        GrantQualification(g$QualificationRequestId, values = s)
    }
    Sys.sleep(30) # wait thirty seconds before polling for new requests
    
    # check HIT status and exit loop if all assignments completed
    h <- HITStatus(annotation = "Random Assignment Experiment")
    x <- sum(as.numeric(h$NumberOfAssignmentsCompleted))
}

The above loop will run until all assignments have been completed. This may take some time, depending on various market factors. When the loop is complete, all assignments are also complete, so we can then access the HIT data.

Step 4. Collect Data

With all assignments completed, it is easy to retrieve the HIT data as a data.frame:

GetAssignments(annotation = "Random Assignment Experiment")