[Air-L] Fw: Re: Missing Data in Qualitative and Online Research

Sun Aug 8 10:54:21 PDT 2010

I'm often frustrated by what I would term a "missing data problem" in
computer-mediated qualitative research.  I study internet-mediated political
organizations in the US (not necessarily web-based, though they all have a
web presence).  I would generally characterize the field as featuring
abundant open data, all of it heavily flawed, and then large swaths of
firewalled data, which usually includes the good stuff.

I'll use political blogs as an example.  The good news is that these provide
you with a time-stamped, readily accessible archive of who said what, to
whom (with hyperlinks), using what frames, exactly when.  Constructing case
studies has gotten a lot easier, and this allows for high quality
comparative case analysis.  The bad news is that what goes into the
networked public sphere through the blogs is only one portion of the
communications around any given case.

I'll be presenting a paper next month at the American Political Science
Association annual meeting that uses the recent US controversy over the
"JournoList" backchannel Google-Group to examine (among other things) the
role that backchannels play among political communities-of-interest.
 Backchannels like JournoList are overwhelmingly common among the US
political netroots.  They provide a forum for networked organizations and
individuals to share information, discuss/debate strategy, vent frustration,
etc.  The communication that occurs on these lists is entirely "missing
data," for three reasons:

1. Restricted access list.  These aren't public lists, they don't show up in
google-group searches.  Membership is provided to people who are invited by
the list moderator.  Moderators are free to establish list rules, which
generally include an off-the-record/do-not-forward requirement.  Mods have
little enforcement power (public shaming, kick members off the list), but it
still makes it impossible for the researcher to find the list unless s/he
meets the right person, and even then the researcher probably only gets
invited on if they agree not to publish anything off of those
communications.

2. The "Fight Club rule."  Most, if not all of, these backchannels include a
rule that list members will not discuss the list in public or to the press.
 JournoList provides a cautionary tale for why this rule is used -- once a
restricted-access list becomes publicly known, it is easy for opponents to
dramatize it as the launching grounds for secret conspiracies.

3. The "Russian Nesting Doll effect."  Since anyone can launch a
Google-Group (it's a 2-screen setup process), once a single list gets large,
individual members will break off and form their OWN also-secret sub-list.
 There are lists within lists, and thanks to the Fight Club rule, it is
technically impossible to determine how many of them there are in total.

A lot of that backchannel communication (like a lot of blog content) is
basically noise.  But some of it is vitally important deliberation or
coordination.  It is, in other words, *communications infrastructure* for
communities-of-interest, mimicking the affordances provided by formal
organizational listservs.  And swimming in the abundance of available blog
data, its easy for us to pretend as though "everything is public/open."
 We're missing huge swaths of communication, both within organizations and
within these networked backchannels.  As a qualitative researcher, that
pretty seriously worries me.  I have a couple of crazy ideas for how to
cobble together a research project on these backchannels, but it'll be six
months at least before it's a solid enough plan to share publicly.

[As someone who dabbles in quantitative research as well, I'll note that the
problems are even bigger there.  So much data, all of it low-quality.  Then
there's plenty of high-quality data, but it's all firewalled/proprietary.  I
could go on a much longer rant about another study I have under way, of
email communications from 70 advocacy groups, and just how much more data
they possess and share with each other than will ever be made publicly
available to you and me.]

...That was a much longer response than I'd intended to provide.  I'm pretty
animated about backchannel lists these days, hope 2 or 3 people found it to
be an interesting tangent.
-Dave

On Sat, Aug 7, 2010 at 4:01 PM, Mohammad H. Hasani <mh_hasani at yahoo.com>wrote:

>
>
> --- On Sat, 8/7/10, Mohammad H. Hasani <mh_hasani at yahoo.com> wrote:
>
> From: Mohammad H. Hasani <mh_hasani at yahoo.com>
> Subject: Re: [Air-L] Missing Data in Qualitative and Online Research
> To: "William Dutton" <william.dutton at oii.ox.ac.uk>
> Date: Saturday, August 7, 2010, 12:59 PM
>
>
>
>
>
> Generally speaking,
>
> 1. I think missing data in a Qual research is data could make
> a significant contribution to the grounded concept or theory but researcher
> neglects or fails to collect. Repeating some phases in data
> collection/analysis,
> conducting parallel Qual research or reverse previewing of the research
> phases seems to help.
>
> 2. The same for CMR, but here, the researcher should also pay more
> attention to the cases; inappropriate cases may cause collecting poor data
> could
> be seen as missed.
>
> Triangulated
> observations may help in this case.
>
>
>
> Mohammad
> H. Hasani,
>
> Instructor
> in Sociology
>
>  Payame Noor
>  University
>
>
>
>
>
>
>
> --- On Fri, 8/6/10, William Dutton <william.dutton at oii.ox.ac.uk> wrote:
>
> From: William Dutton <william.dutton at oii.ox.ac.uk>
> Subject: [Air-L] Missing Data in Qualitative and Online Research
> To: "air-l at listserv.aoir.org" <air-l at listserv.aoir.org>
> Date: Friday, August 6, 2010, 8:17 AM
>
> Dear List Members,
>
> My colleagues and I would greatly appreciate your input to an exploratory
> project on missing data in qualitative and online research.* We would
> appreciate anything from your immediate reactions while reading this e-mail
> to detailed references to literature that has addressed our questions.
>
> There are two very general but heuristically useful questions. Your views
> on either would be welcomed. Feel free to reply privately or to the list, as
> you
>  wish.
>
> 1. What is ‘missing data’ in the context of qualitative research and how is
> it dealt with?
>
> 2. Likewise, in computer-mediated research, are researchers missing
> particular kinds of data, or believe that they are missing particular kinds
> of observations, and how are they compensating or otherwise addressing this
> gap?
>
> Thoughts? Thank you,
>
> Bill
>
> *This is a collaborative project between the ESRC’s National Centre for
> Research Methods (NCRM) ‘hub’ (Graham Crow, Rose Wiles), WISERD (Amanda
> Coffey), Oxford eSocial Science ‘node’ of NCeSS (Bill Dutton, Alison
> Powell), and Qualidata/Timescapes (Libby Bishop), based on our recognition
> of a shared but not well developed problem.
>
> William Dutton, Director
> Professor of Internet Studies
> Oxford Internet Institute
> 1 St Giles', Oxford OX1 3JS  UK
>
> e-mail: william.dutton at oii.ox.ac.uk
> Web: http://www.oii.ox.ac.uk/people/?id=1
> Phone: +44 (0)1865 287 212
> Cell: +44 (0)7768 823906
>
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at:
> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>
>
>
>
>
>
>
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at:
> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/

-- 
Dave Karpf, PhD

Assistant Professor
Journalism and Media Studies Department
School of Communication and Information
Rutgers University, New Brunswick

www.davidkarpf.com
davekarpf at gmail.com