[Air-L] Soliciting opinions about using Amazon's Mechanical Turk (aka Mturk) for survey participant recruitment

[Seda Gurses]

hi,

i believe there are also ethical concerns. i am not sure if this is what you were also asking about but i hope they are nevertheless useful.

on orientalist underpinnings of the concept and labor issues:
“Return of The Crowds: Mechanical Turk and Neoliberal States of Exception” in Digital Labor:The Internet as Playground and Factory. Ed. Trebor Scholz, Routledge, 2012.

last year some researchers raised concerns about anonymity and irb issues related to using amazon mechanical turk:
 Blog post: http://crowdresearch.org/blog/?p=5177
Paper: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2228728

i also remember hearing about an amazon turk workers union that demanded better wages and better working conditions. 
here is a paper on a tool that was designed to support the process:
http://www.ics.uci.edu/~lirani/Irani-Silberman-Turkopticon-camready.pdf
and another article with respect to the problem with the labor conditions created by amazon and other similar crowdsourcing services:
http://economix.blogs.nytimes.com/2013/03/18/the-unregulated-work-of-mechanical-turk/?_r=0

i am not sure if any of the ethical and labor related issues have been addressed over the last months and what ethical questions these raise for using mechanical turk in studies. 
i am sure there are others on this list who have a lot more to say.
good luck!
s.



On Jan 9, 2014, at 10:42 AM, Leticia Bode <lb871 at georgetown.edu> wrote:

> Other articles that address this issue are Berinsky, Huber, and Lenz 2012
> (Political Analysis) and Chandler, Mueller, and Paolacci 2013 (Behavioral
> Research Methods). This is a worthwhile blog post in thinking through some
> of the issues too -
> http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-problems-with-mechanical-turk-stud.html
> .
> Broadly I agree with others that it depends on the question you're asking.
> 
> 
> 
> 
> On Wed, Jan 8, 2014 at 8:09 PM, Tim Muntinga <munt.tim at gmail.com> wrote:
> 
>> Additionally, keep in mind the nature of Mechanical Turk; is there an
>> incentive for the specific group you are targeting to deliver high quality
>> entries? With the amounts of projects executed on a weekly basis, problems
>> of satisficing and routine knowledge of internal tests occur. I find this
>> for surveys quite an important point to consider.
>> 
>> See for more:
>> Kapelner, A., & Chandler, D. (2010). Preventing Satisficing in online
>> surveys.
>> 
>> *----*
>> Tim
>> *W: *www.timmuntinga.com
>> 
>> 
>> 
>> 2014/1/9 Aaron S. Veenstra <aaron at etchouse.com>
>> 
>>> If you're doing experiments, MTurk is great, especially if your likely
>>> alternative is a sample of undergrads. If you're doing surveys, it's
>>> not. The U.S. users are not representative of the American population,
>>> and I'd imagine the same is true of other countries' MTurk
>>> populations. And even if it were, there's no way to prove that to any
>>> kind of satisfaction. Basically, using it for a survey that you want
>>> to generalize to the population isn't getting you a sample that's any
>>> better than posting the survey to Facebook and Twitter and asking
>>> people to spread it around.
>>> 
>>> FWIW, the skew I've seen with MTurk samples compared to the U.S.
>>> population is a) slightly too male, b) slightly too white, c) too
>>> young (though older and with more age variance than an undergrad
>>> sample), d) average income too low, and e) too liberal (though, again,
>>> less skew than I'd expect from undergrads).
>>> 
>>> Aaron
>>> 
>>> On Wed, Jan 8, 2014 at 6:40 PM, Robinson,Cory
>>> <cory.Robinson at colostate.edu> wrote:
>>>> While Qualtrics and SurveyMonkey seem to be generally well respected,
>>> they also come with hefty price tags for participant recruitment ($5-10
>> per
>>> American respondent). Amazon’s Mechanical Turk (aka Mturk), on the other
>>> hand, can recruit participants for far less (<$1/respondent).
>>>> 
>>>> What is the AoIR consensus on utilizing Mturk? I’ve seen articles both
>>> for and against using the service.
>>>> 
>>>> Thanks in advance for all opinions/insights.
>>>> 
>>>> - Cory Robinson
>>>> --
>>>> Stephen Cory Robinson
>>>> cory.robinson at colostate.edu<mailto:cory.robinson at colostate.edu>
>>>> Office: Clark C258A
>>>> http://colostate.academia.edu/StephenCoryRobinson
>>>> 
>>>> _______________________________________________
>>>> The Air-L at listserv.aoir.org mailing list
>>>> is provided by the Association of Internet Researchers http://aoir.org
>>>> Subscribe, change options or unsubscribe at:
>>> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>>>> 
>>>> Join the Association of Internet Researchers:
>>>> http://www.aoir.org/
>>> 
>>> 
>>> 
>>> --
>>> Aaron S. Veenstra
>>> Assistant Professor, Southern Illinois University Carbondale
>>> School of Journalism || 1232 Comm Building
>>> asveenstra at siu.edu || manytoomany.com
>>> _______________________________________________
>>> The Air-L at listserv.aoir.org mailing list
>>> is provided by the Association of Internet Researchers http://aoir.org
>>> Subscribe, change options or unsubscribe at:
>>> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>>> 
>>> Join the Association of Internet Researchers:
>>> http://www.aoir.org/
>>> 
>> _______________________________________________
>> The Air-L at listserv.aoir.org mailing list
>> is provided by the Association of Internet Researchers http://aoir.org
>> Subscribe, change options or unsubscribe at:
>> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>> 
>> Join the Association of Internet Researchers:
>> http://www.aoir.org/
>> 
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
> 
> Join the Association of Internet Researchers:
> http://www.aoir.org/