[Air-L] Air-L Digest, Vol 40, Issue 19

Fri Nov 23 13:38:46 PST 2007

D-train-

Check this out. Don't know if this is up your alley.

-j

On Nov 23, 2007 3:33 PM,  <air-l-request at listserv.aoir.org> wrote:
> Send Air-L mailing list submissions to
>         air-l at listserv.aoir.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
> or, via email, send a message with subject or body 'help' to
>         air-l-request at listserv.aoir.org
>
> You can reach the person managing the list at
>         air-l-owner at listserv.aoir.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Air-L digest..."
>
> Today's Topics:
>
>    1. Coding Analysis Toolkit - A new resource for researchers  with
>       large text annotation tasks (Stuart Shulman)
>
>
> ---------- Forwarded message ----------
> From: "Stuart Shulman" <stuart.shulman at gmail.com>
> To: air-l at listserv.aoir.org
> Date: Fri, 23 Nov 2007 08:37:40 -0500
> Subject: [Air-L] Coding Analysis Toolkit - A new resource for researchers with large text annotation tasks
> Members of the list may be interested in this narrated slide show about the
> Coding Analysis Toolkit developed by QDAP. It is viewable through a web
> browser at:
>
> http://www.qdap.pitt.edu/catshow.htm
>
> The Coding Analysis Toolkit, colloquially known as "CAT", was developed in
> the summer of 2007. It was designed by QDAP Director Dr. Stuart Shulman and
> created in collaboration with Mark Hoy, a Senior Programmer in the Carnegie
> Mellon University School of Computer Science. It is maintained by UCSUR
> Technology Director James Lefcakis. CAT is is hosted on UCSUR servers and
> made available on the web at: http://cat.ucsur.pitt.edu/. The system
> consists of a web-based suite of tools custom built from the ground-up to
> facilitate efficient and effective analysis of text datasets that have been
> coded using the commercial-off-the-shelf package ATLAS.ti (www.atlasti.com).
> We have recently posted a narrated slide show about CAT online.
>
> The Coding Analysis Toolkit was designed to use keystrokes and automation to
> clarify and speed-up the validation or consensus adjudication process.
> Special attention was paid during the design process to the need to
> eliminate the role of the computer mouse, thereby streamlining the physical
> and mental tasks in the coding analysis process. We anticipate that CAT will
> open new avenues for researchers interested in measuring and accurately
> reporting coder validity and reliability, as well as for those practicing
> consensus-based adjudication. The availability of CAT can improve the
> practice of qualitative data analysis at the University of Pittsburgh and
> beyond.
>
> Currently about 50 beta testers located in several countries have accounts
> on CAT. They have been given free access to the system for the rest of the
> calendar year. Systematic user feedback will be gathered via a beta tester
> web survey and will shape the future development of CAT. The capabilities of
> CAT and its reliability as a software tool may be sufficiently robust to
> merit commercial licensing to users starting in 2008. The CAT system allows
> a user to register for an account to log on, upload exported coded results
> from ATLAS.ti into the system, and run comparisons of inter-rater
> reliability measured using Fleiss' Kappa and Krippendorff's Alpha. The user
> can also choose to perform a code-by-code comparison of the data, revealing
> tables of quotations where coders agree, disagree, or overlap. For any
> comparisons, the user can view the data on the screen, or alternatively,
> download the data file as a rich-text file (.rtf).
>
> CAT's core functionality allows for the adjudication of coded items by an
> "expert" user who is a sub-account attached to the primary account holder of
> the system. The website and database itself resides on a Windows 2003 UCSUR
> server and the programming for the website is done using HTML, ASP.net
> 2.0and JavaScript. An expert user can log onto the system to validate
> the codes
> assigned to items in a dataset. While the expert user is validating codes,
> the system also keeps track of which codes are valid and which coders
> assigned those codes. This information is used to keep a historical track
> record of coders for assessing coder accuracy over time. It also allows the
> account holder to see a rank order list of the coders most likely to produce
> valid observations, report the overall validity scores by code, coder, or
> entire project, and end up with a 'clean' dataset consisting of only valid
> observations.
>
> In a newly developed CAT module, which is being beta tested internally
> during the fall 2007 semester by five QDAP coders, the project manager is
> able to upload raw datasets and have users code those datasets directly
> through the CAT interface. As is the case with the original adjudication
> toolkit, this new module features automated loading of discrete quotations
> and requires only keystrokes, instead of mouse clicks and drags, to apply
> the codes to the text. We estimate coding tasks using CAT are completed 2-3
> times as fast as identical coding tasks conducted using ATLAS. While this
> high-speed "mouse-less" coding module would poorly serve many traditional
> qualitative research approaches, it is ideally suited to annotation tasks
> routinely generated by computer scientists.
>
> ~Stu
>
>
>
> --
> Dr. Stuart W. Shulman
> Director, Sara Fine Institute
> School of Information Sciences
> Director, Qualitative Data Analysis Program
> University Center for Social and Urban Research
> http://qdap.ucsur.pitt.edu
> University of Pittsburgh
> 121 University Place, Suite 600
> Pittsburgh, PA 15260
> 412.624.3776 (v) 412.624.4810 (f)
> http://shulman.ucsur.pitt.edu
> Editor, Journal of Information Technology and Politics
> http://www.jitp.net
>
>
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>

-- 
Jason Gallo

PhD Candidate
Media, Technology and Society Program
School of Communication
Northwestern University

Project Coordinator
WebUse Project
http://www.webuse.northwestern.edu/