Working the Crowd
posted by Miriam Cherry
I’ve written before about crowdsourcing and its accompanying employment law issues. And so when I saw this recent article from the Economist, I thought that it encapsulated both the potential of virtual work as well as its downside. The article notes that psychologists are using the Mechanical Turk to broaden their psychological experiments so that they are truly global and representative of more than the perspectives of western college students. Sounds great! Er, well. Knowing how hard it is to earn minimum wage on the Mechanical Turk (because my research assistant and I have tried to earn it, and failed to do so on numerous occasions), it becomes a darker story. Here are some excerpts from the article:
Crowdsourcing is a way to get jobs like deciphering images, ranking websites and answering surveys done for money by online workers. Several firms offer the service, including oDesk, CrowdFlower and Elance. But by far the most popular for scientific purposes is Mechanical Turk, which is run by Amazon and is named after an 18th-century chess-playing machine in which a human secretly moved the pieces. Mechanical Turk has more than 500,000 people, known as Turkers, in its workforce. For the hard-pressed, cash-strapped psychologist, this is a godsend. Turkers, despite the fact that half of them have at least one degree, are willing to work for peanuts. (Their median wage is about $1.40 an hour.) Most, indeed, seem to regard the tasks they are set as more like a paying hobby than an actual job. And, crucially, they are growing more cosmopolitan with each passing year. Though 40% are still from America, a third are Indian and the rest come from about 100 other countries. . . .using Turkers instead of undergrads does offer some genuine diversity.
One researcher who has taken advantage of that diversity is David Rand, a lecturer in psychology at Harvard University. He is using Mechanical Turk to reconsider the results of several experiments originally conducted mainly on students. In a recent study of moral decision-making, for example, he recruited hundreds of Turkers to repeat a classic thought experiment known as the trolley problem. This confronts its participants with a dilemma—a runaway railway trolley will kill a group of people unless the subject of the study chooses to push a single individual in front of it, in order to slow it down. Doing so will kill that individual, so the dilemma is whether to kill one person deliberately, or several through inaction.… Dr Rand is re-running Dr Herrmann’s experiments on Mechanical Turk—at a tenth of the cost of the original work….
Questions of ethics have also arisen. Some people think research projects which pay wages of less than $2 an hour are exploitative—even though that is the going rate for other Turker activities. Conversely, according to Karen Fort, of France’s Institute of Scientific and Technical Information, at least one university has already prohibited the use of grant funds for this sort of study, for fear that Turkers could claim status as employees. For many researchers, though, the appeals of crowdsourcing—bargain prices, vast supply and enormous scale—are too attractive to ignore. Indeed, the new methodology might democratise the very practice of psychology, allowing those without a laboratory or university behind them to join in as well.
I have a very strong view on this. I believe that in its current form, the relations between workers and the psychologists described here are exploitative. Beyond that, there is a strong argument that employment on these terms violates the Fair Labor Standards Act. Researchers and their universities should be ashamed. It is ironic that someone ostensibly investigating ethics around the world cannot do a better job when designing his own experiment. Further questions: How is this getting past university IRBs? Has anyone at the Department of Labor taken notice of this? And where are the enterprising wage-and-hour labor attorneys? (The spokewoman for France’s Institute of Scientific and Technical Information has a point, in my view).
[Hat-tip: Jeffrey Harrison]
July 18, 2012 at 7:10 pm
Posted in: Uncategorized
Print This Post
Responses (10)
Michelle Meyer - July 18, 2012 at 8:33 pm
There is a vast literature on the ethics of paying research subjects in all kinds of situations, with strong views on both sides. The primary (and very common) objection to paying them is that doing so is coercive. Research participation is supposed to be both informed and voluntary, and the concern among both research ethicists and IRBs is that repaying out-of-pocket costs is fine, but offering research participation as a way to profit — or, to put it another way, make a living, or part of one — undermine voluntariness, perhaps to the point of making the research offer coercive. And so, when IRBs conduct risk-benefit analyses of proposed studies, they don’t count any payment the researcher plans to offer as a benefit to subjects, and indeed, they will usually object to such payment if it seems to them to be “too much” — and the poorer the subjects are expected to be, the more coercive the offer. One *might* make such an argument in the context of, say, an especially risky trial. But IRBs have extended this payment-as-coercion “logic” to all manner of studies. (Oddly, the conventional wisdom of research ethicists who consider research in the developing world is that all manner of compensation in-kind of obligatory (e.g., ancillary medical care, post-trial access to an experimental drug), even though such in-kind compensation might be just as “coercive” as cash.)
As you suggest, avoiding “coercion” by paying little or nothing may come at the expense of ensuring exploitation, which IRBs and research ethicists also worry a lot about. I wouldn’t, however, assume that any subject, including any poor or otherwise disadvantaged subject, who is not paid well — or, indeed, at all — is necessarily exploited. First, subjects in developing countries who are paid a pittance by U.S. minimum wage standards might be doing okay in their own countries.
Second, the Belmont Report describes research subjects as, “in essence, volunteers.” Clearly, people participate in research for all kinds of reasons. Some of them quite explicitly do it for the money, earning up to 40K/year by participating in various clinical trials in the U.S. (more, if they flout the rules). In these cases, it may make a lot of sense to treat this as work and them as workers. (For more on that argument, see Holly Lynch Fernandez’s talk, available here: http://www.law.harvard.edu/programs/petrie-flom/events/conferences/hsr.html) But if the Belmont Report’s prescriptive view of research participation as volunteer work doesn’t always match reality, neither, I think, does the view that research participation can never be unpaid volunteer work. To conclude that low- or no-pay research offers to poor people is necessarily exploitative is to assume that poor and otherwise disadvantaged people are incapable of being altruistic, or that they should never be permitted to be altruistic (even, as here, where the research may be time-consuming and boring, but is very unlikely to be physically, psychologically or legally risky).
And in fact there are many other motivations for participating in research beyond altruism and money, including curiosity, adventure-seeking, and a sense of moral obligation (somewhat different from at least some altruism, which might yield warm glow utility for the subject). Here’s a review of several studies of motivations for participating in Mechanical Turk in particular, which suggests that Turkers are, in fact, heterogeneous in their motivations: http://schader.bwl.uni-mannheim.de/fileadmin/files/publikationen/Kaufmann_Schulze_Veit_2011_-_More_than_fun_and_money_Worker_motivation_in_Crowdsourcing_-_A_Study_on_Mechanical_Turk_AMCIS_2011.pdf
Michelle Meyer - July 18, 2012 at 8:36 pm
Oops, that should be Holly Fernandez Lynch. (Sorry, Holly!)
A.J. Sutter - July 18, 2012 at 8:49 pm
Very nice point about the irony of these methods being used in ethics research. The whole crowdsourcing field might do with some rhetorical deconstruction. E.g., “crowdsourcing” as an Orwellian (1984-ish) euphemistic neologism (though with the mixed reputation of “outsourcing” these days, the “-sourcing” morpheme could be a tip-off that the practice might not be as beneficial as it’s touted to be), the Mechanical Turk metaphor as suggesting dehumanized and anonymized workers, and, not least, the defensive deployment of “democratization” at the end of the quote. Though of course Athenian democracy was based on slave labor, too.
dave hoffman - July 18, 2012 at 9:13 pm
Miriam
On the IRB question alone, have you considered whether the relevant regulations in fact permit IRBs to prohibit, or even substantially regulate, surveys? Should they? There is, as Michelle points out, a vast literature on this. IRB mission overcreep isn’t something to lightly accept!
Dan Culley - July 18, 2012 at 11:11 pm
Better to deny them the opportunity to make what little money they can, so they have nothing, then to pay them $2/hour to answer a few questions about trolleys killing people and have you feel bad about it?
This isn’t experimental medical research — it’d be one thing if we were talking about exposing these people to significant danger and had reason to believe they may not appreciate the risk. Absent some reason to believe they don’t know what they are getting into, these people must be better off taking the opportunity than not, or they wouldn’t be taking it.
Ken Arromdee - July 19, 2012 at 10:25 am
Better to deny them the opportunity to make what little money they can, so they have nothing, then to pay them $2/hour to answer a few questions about trolleys killing people and have you feel bad about it?
That argument can be made for all minimum wage laws: it’s better for them to work below minimum wage than to work for nothing.
Some people do oppose all minimum wage laws, but do you want to commit to that position?
Miriam A. Cherry - July 19, 2012 at 12:19 pm
@Michelle: Mturk bills itself as “a marketplace for work.” While workers may participate on the Mturk for a variety of reasons, that doesn’t negate that the participants are working. (I like being a professor; I receieve hedonic benefits from my job, but that doesn’t mean that it isn’t still a job). In other words, we don’t look to someone’s enjoyment (or lack thereof) and then decide if they are really employed or not.
Could crowdsourced work be voluntary? Maybe. But in my view the Mturk isn’t set up for volunteers. You are told how much you will be paid for a task. You perform the task.
Some number of Turkers are in the US – so it would fall below our minimum wage – and I think there are some serious ethical questions about using third world labor at $1.40 an hour – even if it is better than subsistance farming, is there any informed consent? Or, as my judge used to put it, does this pass the smell test?
@Dave: Good point. I wasn’t really speculting too much about IRBs qua IRBs, I just wondered where the watchdogs were on this.
@Dan: @Ken has it. Your argument is the same as those who are opposed all minimum wage laws.
Dan Culley - July 26, 2012 at 9:05 am
Just saying the argument is the same as those who are opposed to all minimum wage laws isn’t actually a response. (What happens exactly, if I “commit” to that position?) Why do you think that the turkers would be better off if you didn’t use them? Overall, I do think minimum wage laws do more harm than good, though in certain situations I recognize that they can move the terms of trade in an efficient way. I just have serious doubts about that being the case here — for most of the people turking, this is just a supplement to income and so their labor supply is more elastic, and it seems very likely that the low cost here is leading to much more research, and so more employment.
I also don’t really understand the fetishism over “informed consent” with answering anonymous survey questions. What danger are the participants being placed in exactly? Why would we think that they are not better off as a result of their choice to participate?
Jacqueline Parks - August 25, 2012 at 8:00 am
I am a Mechanical Turk worker. I am American, and our family has found ourselves in a hard place financially. I work at Turk for about 12 hours a day, and my average pay is $1.40 an hour. (How odd that it is exactly the median pay!) I keep track because I was told that I could make more, and I keep hoping to find a way.
I keep working because we need money to be OK right now. I can’t wait until some unknown future date when I might find a better job. Matter of fact, we are at risk for having our electricity shut off, and I need $179 fast and am hoping to have that total deposited in my account soon.
I have been working for Mechanical Turk for just under a month. At first, I was excited to be able to earn money easily from home. I would be sad if researchers left because that would lower my income. Then I would feel more hopeless.
At the same time, I do feel like a sweat shop employee. I do not make minimum wage. I work really hard. Many of the Requesters are rude in the things they write on their request pages. My favorite are the researchers who insinuate that Turkers are trying to get paid without doing good jobs. We are getting paid a pittance, some working long hours, and we do the best we can. Not all are like this, but it seems to be a growing thing to use a threatening tone instead of a positive encouraging tone when requesting that people pay attention and do a good job.
I also feel somewhat trapped. I have to keep working at Turk to get the 16 or 17 dollars deposited into my bank account each day. This leaves me no time to find other money earning opportunities. I do not know if using Mechanical Turk for research or other crowd sourcing is ethical or not. I clearly see the dichotomy of not enough pay and yet not wanting this small amount of income to lessen or disappear. Just thinking about it is a source of anxiety.
I would express my desire that the contractors at least remember that those doing work this way are very low paid but still people and to treat them with respect both with regards to writing tone and with regards to paying as high a rate as possible.
Ellen M. Doye - November 29, 2012 at 11:06 am
A lawsuit challenging CrowdFlower’s failure to pay minimum wage under the Fair Labor Standards Act to its crowdsourced U.S. work force was filed in the United States District Court for the Northern District of California by Christopher Otey in Civil Action No 12-5524 on October 26, 2012.
Leave a Reply