[Air-L] Let's talk about AoIR.

[Kathleen Fitzpatrick]

Hi, all. I wanted to mention that another of the organizations I'm active
in, the Alliance of Digital Humanities Organizations, which puts on the
annual DH conference, recently experienced some growing pains around its
conference submission and reviewing process. The conference has recently
seen a dramatic uptick in submissions, from a much more diverse range of
fields, methodologies, and parts of the world than in the past, and as a
result recent conference program processes were beset by challenges in
getting the right reviewers for the right proposals. And this produced a
lot of hard feelings across the board: among authors who felt rudely
treated, among reviewers who felt overtasked, and among program committee
members who totally *were* overtasked.

This year's program committee, led by Bethany Nowviskie, introduced a
number of changes in the process, which Bethany detailed in a blog post:
http://nowviskie.org/2012/cats-and-ships/. The most important of these
changes may have been the bidding process, in which reviewers get to
request particular abstracts to review (as well as marking those abstracts
for which they are not qualified). I have served as a reviewer for the last
several years and can say that the proposals I was asked to review this
year were far more appropriate to my subfield than they had ever before
been.

The process also included a few other crucial changes: First, after a
reviewer submitted her reviews, she was able to see the other reviews of
those abstracts (though without reviewer names attached), and she could
modify or add to her reviews in response. Second, after the review period
was closed, authors were given the opportunity to respond to the reviews.
And finally, the program committee was able to return to particular
reviewers to ask them for clarification or reconsideration, before making a
final decision.

All of this, from what I've heard, made the committee's process more
complex, but I did not see any complaints online about the process or its
results this year, while recent years had produced lots of audible
discontent. Bethany has promised to write some more assessing the results
of the process; the conference is coming up, so perhaps that will be
available soon.

All best,
Kathleen

--
Kathleen Fitzpatrick // Director of Scholarly Communication
Modern Language Association // mla.org // @kfitz


On Thu, May 30, 2013 at 2:00 PM, Holly Kruse <holly.kruse at gmail.com> wrote:

> I think that it's great that we're having this discusson on the list. I've
> reviewed for the past eight or nine AoIR conferences, and I like to think
> that I have a pretty good handle on reviewing for this conference. I could
> be wrong. Still, I propose that for the next conference we craft clear
> guidelines for reviewers, so that reviewers are less apt, for instance, to
> expect a roundtable abstract to demonstrate the same level of theoretical
> development as paper proposals for a panel. Likewise, we could do our best
> to ensure that if we keep the SPIR template for submissions, reviewers are
> aware that not including a methodology or results section is totally fine
> if it's not appropriate for the paper that's being proposed. I'm willing to
> spearhead this effort at clarification.
>
> Holly
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at:
> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>