It’s not uncommon for scientists to use Amazon’s Mechanical Turk for research — they can quickly gather survey data in return for a small outlay to participant…
It’s not uncommon for scientists to use Amazon’s Mechanical Turk for research — they can quickly gather survey data in return for a small outlay to participants. Academics are quickly discovering the limits of the task service with the modern bot-happy internet, however. Numerous psychology researchers have reported a sharp rise in the number of junk responses to their questions, many of which appear to come from bots. The offenders not only spam nonsensical answers to questions (such as «nice» or «good»), but tend to originate from the same location. And while anti-bot measures like captchas are in use, they aren’t always reliable.
There’s a chance that these could be people copying and pasting text, or using some form of automation to find the surveys (which Amazon allows). However, Amazon wouldn’t provide evidence one way or the other. The company told Wired in a statement that it had «automated and manual mechanisms» to fight bots, but it wouldn’t comment on bot activity levels or specific incidents.
Whatever the cause, the psychology study problems underscore deeper issues with Mechanical Turk. Bots are a reality on the service, and Amazon doesn’t always appear to acknowledge them even after multiple reports. And then there’s Mechanical Turk’s low per-task pay. While that can help scientists worried that large rewards could skew results, it encourages completing tasks en masse rather than providing high-quality answers. It may be increasingly difficult to conduct meaningful surveys on Amazon’s service, at least without tossing out many of the responses.
