Mechanical Turk, Research Ethics, and Research Assistants
posted by Glenn Cohen
A recent faculty workshop by my witty and brilliant colleague Jonathan Zittrain on “ubiquitous human computing,” (this youtube video captures in a different form what he was talking about ), prompted me to thinking about some ways in which platforms like Amazon’s Mechanical Turk, interface with university research and research ethics in interesting ways.
For those unfamiliar, Mechanical Turk allows you to farm out a variety of small tasks (label this image, enter date of this .pdf to a spreadsheet, take a photo of yourself with the sign “will turk for food,” etc) at a price per unit you set. Millions of anonymous users can then do the task for you and collect the bounty, a form of microwork.
As Jonathan detailed, this raises a host of fascinating issues, but I want to focus on two that are closer to bioethics.
First, I have begun to see some legal academics recruiting populations for experimental work using Mechanical Turk, and there is an emerging literature on the pros and cons of subject recruitment from these populations. Are Mechanical Turkers “research subjects” within the legal (primarily the Common Rule if one receives federal funding) or broader ethical sense of the term? Should they be? Take as a tangible example the implicit bias research of the kind Mahzarin R. Banarji has made famous, and imagine it was done over something like Mechanical Turk. How (if at all) should the anonymity of the subject, the lack of subject-experimenter relationship of any sort, the piecemeal nature of the task, etc, change the way an institutional review board reviews the research? It is a mantra in the research ethics community that informed consent is supposed to be a “process” not a document, but how can that process take place in this anonymous static cyberspace environment?
Second, consider research assistance.
I often have my R.A.’s read over papers before I send them out to hunt for typos (alongside more substantive tasks I give them). Imagine that tomorrow I decided (imagine a shrinking research budget due to times of fiscal austerity) to farm the typo-hunt off to Mechanical Turk because I could get results faster and at one tenth the price, since there were individuals in destitute circumstances willing to do it at a rate far below that I pay my (wonderful, in case they are reading) R.A.’s. Even if the accuracy of the Turkers was individually less good, it seems plausible that having four of them pour over each page might be better and still cheaper than using R.A.’s to do it. Lest you think this only an interesting hypothetical, consider Samasource, whose mission statement suggests it “enables marginalized people, from refugees in Kenya to women in rural Pakistan, to receive life-changing work opportunities via the Internet” in just this way.
Would I have violated any rules at your university? Have I done something wrong? Perhaps I have deprived Harvard students of the opportunity to work closely with a faculty member (although on typo hunting?) Am I problematically circumventing Harvard’s minimum wage for R.A. work? Am I exploiting these Kenyan refugees or rural Pakistani women or instead giving them “life-changing work opportunities via the Internet?”
I’d be curious to hear the thoughts of any readers, as well as any reports on whether your institution has a policy on this subject.
August 3, 2010 at 9:49 am
Posted in: Amazon, Anonymity, Bioethics, Bright Ideas, Google & Search Engines, Law and Psychology, Law School, Law School (Scholarship), Technology, Web 2.0
Print This Post
Responses (5)
David Zaring - August 3, 2010 at 8:52 pm
Huh – my sense, based on chats with experimenters around my university, is that Mechanical Turk is closing on becoming 50% a venue for social research. Not sure how they got the relevant HSR approvals, but I think they’ve done it in spades. So you’ll have company if you do it.
John Horton - August 4, 2010 at 10:14 pm
There’s been quite a bit of discussion about these issues
in various places.
On the ethics/legal aspects of crowdsourcing, there is
Alek Felsteiner’s paper:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1593853
As well as his post on the Crowdflower blog focusing on regulation (as well as some back-and -forth in the comments):
http://blog.crowdflower.com/2010/06/regulating-distributed-work-part-three-why-its-a-good-idea/
There was a debate at HCOMP 2010 on fairness of crowdsourcing (stimulated by a paper by Six Silberman et al.):
http://behind-the-enemy-lines.blogspot.com/2010/07/liveblogging-from-hcomp-2010.html
Here’s my own little paper investigating what Turkers seem to think of their employers:
http://www.technologyreview.com/blog/arxiv/24646/
Regarding experiments, I also have a paper similar to the one you linked to, re: conducting experiments online:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1591202
Anecdotally, there seems to be lots of variance in what different universities require to gain IRB approval.
Jeremy A. Blumenthal - August 17, 2010 at 2:15 pm
Do you see MTurk as different from other online or internet-based data collection methods (like Mahzarin’s implicit bias website or others)? Over the last 15 years or so such internet-based methods have become common in a number of social science disciplines. As a result there’s been much discussion on both ethics and practice–how to ensure informed consent, how to ensure privacy, how to ensure data quality, whether respondents differ from in-person respondents, etc. (Here’s one of many examples: http://jrp.icaap.org/index.php/jrp/article/view/30/51.)
More broadly, though, I don’t think there is (or should be) a question that such respondents are research subjects within an IRB’s purview. Not all researchers know or think to get IRB approval, though, which is a separate issue….
Broken Turk - November 1, 2010 at 11:39 pm
Mechanical Turk is a phenomenal example of the power of the internet to leverage all of the world’s intelligence and human processing surpluses. Massive untapped resources. However, Mechanical Turk’s technological potential has advanced faster than its systems for managing users (workers), and a look through the forums where Turkers talk shop reveals an embarrassing litany of frustrations, sudden blocks and bans, uncertainties, and customer services incidents that Amazon would never stand for in its flagship site. I’ve documented the most poignant of these criticisms at a blog called Broken Turk (http://brokenturk.blogspot.com)
FuckYouTurker - March 7, 2013 at 2:21 pm
The science you idiots are generating after fucking over your participants with horseshit pay in order to further your careers is a joke. You put up 15 pages of questionnaires wanting authentic responses after shitting in the face of you participants by paying them 5 or 10 cents. You pay garbage, you get garbage, and your results or laughable horseshit. I’d throw any finding from turk subjects in the trash. Why should YOU further your career at the expense of bad science and exploitation. Fuck you people.
Leave a Reply