[Air-L] Fw: Re: Missing Data in Qualitative and Online Research

Jessica L. Beyer jlbeyer at u.washington.edu
Sun Aug 8 14:32:28 PDT 2010


I’d like to second Dave’s comments.  My research concerns political behavior in/arising from anonymous social locations online with very large user bases (1 million+).  It’s a different context with many of the same issues.

The illusion of massive amounts of open data:

One of the beautiful/horrible things about researching anything online is it is like standing in front of a firehose.  The amount of data seems to be enormous, but in line with what Dave stated, often the “good” data—-or in my cases-—very important additional data are not visible without passing through gatekeepers.  In the cases of private companies, the gatekeepers are often unwilling to provide this data for proprietary reasons.  In the cases of anonymous spaces, it can be difficult to even find who the gatekeepers/moderators/admins are in the masses to inquire about the things you cannot see.  And, in the case of splinter communities such as those Dave mentioned-—unless you are part of their “in-group” you may never find yourself invited to view their conversations (or even know they exist).  Frustratingly, many of these splinter groups can have huge effects on online spaces-—and 
be engaging in really weird and interesting behavior that has some serious theoretical importance. (Further, to even be invited/know they exist you'd have to be engaging in participant-observation research--which might not be appropriate/preferable for all researchers.)

The ephemeral nature of the data:

We think of things on the internet as living forever (photos destroying your chances of employment forever!)—-but in many spaces it does not.  Some online spaces are constructed in such a way that information disappears quickly and is difficult/impossible to re-acquire (ex: 4chan, temporary sites to organize protest, etc.) and the information that is left behind is not necessarily to be trusted as there is an awareness that “outsiders” will see it (ex: Encyclopedia Dramatica).  Others-—and this feeds into point one-—have a regulation system in which posts just disappear without note or comment.  (Friends who study authority online, particularly religion, find this to be challenging.) And, others involve real time conversations of which there is no record-—such as IRC channels.  Meaning, if you aren’t  watching 24/7 you are bound to miss something that could be important-—and 
later must go back and attempt to reconstruct it.  And, reconstructing can be difficult even if there is a record because it’s possible to find the same event retold with small variations across sources.

Anyway, thank you Dave for your post.

Best to all,
Jessica

--------------------------------------
Jessica L. Beyer
Doctoral Candidate
University of Washington
Department of Political Science
http://students.washington.edu/jlbeyer
--------------------------------------

On Sun, 8 Aug 2010, Dave Karpf wrote:

> I'm often frustrated by what I would term a "missing data problem" in
> computer-mediated qualitative research.  I study internet-mediated political
> organizations in the US (not necessarily web-based, though they all have a
> web presence).  I would generally characterize the field as featuring
> abundant open data, all of it heavily flawed, and then large swaths of
> firewalled data, which usually includes the good stuff.
>
> I'll use political blogs as an example.  The good news is that these provide
> you with a time-stamped, readily accessible archive of who said what, to
> whom (with hyperlinks), using what frames, exactly when.  Constructing case
> studies has gotten a lot easier, and this allows for high quality
> comparative case analysis.  The bad news is that what goes into the
> networked public sphere through the blogs is only one portion of the
> communications around any given case.
>
> I'll be presenting a paper next month at the American Political Science
> Association annual meeting that uses the recent US controversy over the
> "JournoList" backchannel Google-Group to examine (among other things) the
> role that backchannels play among political communities-of-interest.
> Backchannels like JournoList are overwhelmingly common among the US
> political netroots.  They provide a forum for networked organizations and
> individuals to share information, discuss/debate strategy, vent frustration,
> etc.  The communication that occurs on these lists is entirely "missing
> data," for three reasons:
>
> 1. Restricted access list.  These aren't public lists, they don't show up in
> google-group searches.  Membership is provided to people who are invited by
> the list moderator.  Moderators are free to establish list rules, which
> generally include an off-the-record/do-not-forward requirement.  Mods have
> little enforcement power (public shaming, kick members off the list), but it
> still makes it impossible for the researcher to find the list unless s/he
> meets the right person, and even then the researcher probably only gets
> invited on if they agree not to publish anything off of those
> communications.
>
> 2. The "Fight Club rule."  Most, if not all of, these backchannels include a
> rule that list members will not discuss the list in public or to the press.
> JournoList provides a cautionary tale for why this rule is used -- once a
> restricted-access list becomes publicly known, it is easy for opponents to
> dramatize it as the launching grounds for secret conspiracies.
>
> 3. The "Russian Nesting Doll effect."  Since anyone can launch a
> Google-Group (it's a 2-screen setup process), once a single list gets large,
> individual members will break off and form their OWN also-secret sub-list.
> There are lists within lists, and thanks to the Fight Club rule, it is
> technically impossible to determine how many of them there are in total.
>
> A lot of that backchannel communication (like a lot of blog content) is
> basically noise.  But some of it is vitally important deliberation or
> coordination.  It is, in other words, *communications infrastructure* for
> communities-of-interest, mimicking the affordances provided by formal
> organizational listservs.  And swimming in the abundance of available blog
> data, its easy for us to pretend as though "everything is public/open."
> We're missing huge swaths of communication, both within organizations and
> within these networked backchannels.  As a qualitative researcher, that
> pretty seriously worries me.  I have a couple of crazy ideas for how to
> cobble together a research project on these backchannels, but it'll be six
> months at least before it's a solid enough plan to share publicly.
>
> [As someone who dabbles in quantitative research as well, I'll note that the
> problems are even bigger there.  So much data, all of it low-quality.  Then
> there's plenty of high-quality data, but it's all firewalled/proprietary.  I
> could go on a much longer rant about another study I have under way, of
> email communications from 70 advocacy groups, and just how much more data
> they possess and share with each other than will ever be made publicly
> available to you and me.]
>
> ...That was a much longer response than I'd intended to provide.  I'm pretty
> animated about backchannel lists these days, hope 2 or 3 people found it to
> be an interesting tangent.
> -Dave
>
> On Sat, Aug 7, 2010 at 4:01 PM, Mohammad H. Hasani <mh_hasani at yahoo.com>wrote:
>
>>
>>
>> --- On Sat, 8/7/10, Mohammad H. Hasani <mh_hasani at yahoo.com> wrote:
>>
>> From: Mohammad H. Hasani <mh_hasani at yahoo.com>
>> Subject: Re: [Air-L] Missing Data in Qualitative and Online Research
>> To: "William Dutton" <william.dutton at oii.ox.ac.uk>
>> Date: Saturday, August 7, 2010, 12:59 PM
>>
>>
>>
>>
>>
>> Generally speaking,
>>
>> 1. I think missing data in a Qual research is data could make
>> a significant contribution to the grounded concept or theory but researcher
>> neglects or fails to collect. Repeating some phases in data
>> collection/analysis,
>> conducting parallel Qual research or reverse previewing of the research
>> phases seems to help.
>>
>> 2. The same for CMR, but here, the researcher should also pay more
>> attention to the cases; inappropriate cases may cause collecting poor data
>> could
>> be seen as missed.
>>
>> Triangulated
>> observations may help in this case.
>>
>>
>>
>> Mohammad
>> H. Hasani,
>>
>> Instructor
>> in Sociology
>>
>>  Payame Noor
>>  University
>>
>>
>>
>>
>>
>>
>>
>> --- On Fri, 8/6/10, William Dutton <william.dutton at oii.ox.ac.uk> wrote:
>>
>> From: William Dutton <william.dutton at oii.ox.ac.uk>
>> Subject: [Air-L] Missing Data in Qualitative and Online Research
>> To: "air-l at listserv.aoir.org" <air-l at listserv.aoir.org>
>> Date: Friday, August 6, 2010, 8:17 AM
>>
>> Dear List Members,
>>
>> My colleagues and I would greatly appreciate your input to an exploratory
>> project on missing data in qualitative and online research.* We would
>> appreciate anything from your immediate reactions while reading this e-mail
>> to detailed references to literature that has addressed our questions.
>>
>> There are two very general but heuristically useful questions. Your views
>> on either would be welcomed. Feel free to reply privately or to the list, as
>> you
>>  wish.
>>
>> 1. What is ‘missing data’ in the context of qualitative research and how is
>> it dealt with?
>>
>> 2. Likewise, in computer-mediated research, are researchers missing
>> particular kinds of data, or believe that they are missing particular kinds
>> of observations, and how are they compensating or otherwise addressing this
>> gap?
>>
>> Thoughts? Thank you,
>>
>> Bill
>>
>> *This is a collaborative project between the ESRC’s National Centre for
>> Research Methods (NCRM) ‘hub’ (Graham Crow, Rose Wiles), WISERD (Amanda
>> Coffey), Oxford eSocial Science ‘node’ of NCeSS (Bill Dutton, Alison
>> Powell), and Qualidata/Timescapes (Libby Bishop), based on our recognition
>> of a shared but not well developed problem.
>>
>> William Dutton, Director
>> Professor of Internet Studies
>> Oxford Internet Institute
>> 1 St Giles', Oxford OX1 3JS  UK
>>
>> e-mail: william.dutton at oii.ox.ac.uk
>> Web: http://www.oii.ox.ac.uk/people/?id=1
>> Phone: +44 (0)1865 287 212
>> Cell: +44 (0)7768 823906
>>
>> _______________________________________________
>> The Air-L at listserv.aoir.org mailing list
>> is provided by the Association of Internet Researchers http://aoir.org
>> Subscribe, change options or unsubscribe at:
>> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>>
>> Join the Association of Internet Researchers:
>> http://www.aoir.org/
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> The Air-L at listserv.aoir.org mailing list
>> is provided by the Association of Internet Researchers http://aoir.org
>> Subscribe, change options or unsubscribe at:
>> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>>
>> Join the Association of Internet Researchers:
>> http://www.aoir.org/
>
>
>
>
> -- 
> Dave Karpf, PhD
>
> Assistant Professor
> Journalism and Media Studies Department
> School of Communication and Information
> Rutgers University, New Brunswick
>
> www.davidkarpf.com
> davekarpf at gmail.com
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>


More information about the Air-L mailing list