[Air-L] Request: Stop sending reviewer scores to authors

[Tama Leaver]

Dear Jill,

I'm sorry to hear receiving review scores was a bad experience for you.

Thanks for your suggestion about blinding the scores themselves and only
providing comments, we will take it on board as part of the discussion and
planning for next year's conference.

Since you asked I should add, though, that there are at least two functions
currently achieved by having scores visible.

One is simple transparency: we do use the scores provided as part of the
review process. It isn't the only signal, and this year there was a larger
team so we could spend more time focusing on the qualitative comments in
much more detail, but it is part of the review process and people often
feel uncomfortable that they can't see part of what is being used to judge
their submission. And of course, in spite of the many hours the conference
committee spends on organising and supervising the review process,
sometimes a bad review still slips through the cracks. The transparency of
providing the review scores enables our participants to dispute them, if
they feel a score is inappropriate.

(Just from personal experience, I hate that our major grants don't show the
actual scores, just the comments, which clearly sometimes don't align well
at all with the comments people make.)

The second reason is social: seeing our own numbers (whether fair or
otherwise) is part of the process by which reviewers calibrate what fair
numbers look like. While we do provide guidance on what the numbers
indicate, sometimes your own past scores are one of the best ways to help
people think about the numbers they provide in their reviews.

Those two points aside, I would also add, there is an internal conversation
already underway about potentially moving away from a 1 to 100 scale to
something else, and this feedback will definitely be considered in that
discussion.  If you have other suggestions of a productive reviewing system
/ review communication system that you’ve experienced elsewhere and think
would be manageable for our organization, being mindful of the scale of our
conference and that we stretch across many disciplines including
quantitative ones, please do feel free to share your ideas with us.

Many thanks,
Tama

On Thu, Jun 17, 2021 at 2:19 PM Jill Walker Rettberg <
Jill.Walker.Rettberg at uib.no> wrote:

> I would like to make a formal request to the AoIR board that for future
> conferences, the organization NOT send out reviewer scores to authors.
>
> I get why scores are useful for the selection process but it’s hard to see
> how they are helpful for the author, and easy to see how they can be
> harmful and demotivating. I’m in a secure position and can handle bad
> reviews, but if I had received the scores I received this year as a junior
> scholar I would have been devastated.
>
> The reviewer comments are useful to authors – thank you to all the
> reviewers for the hard work, and for many constructive comments that will
> be helpful for authors developing their ideas. But there is no reason to
> send the scores to authors.
>
> There is also so much research showing the arbitrariness of peer review
> (it’s not random but there is a LOT of subjectivity involved) that putting
> such apparently objective scores out is really misleading. I mean, even
> national and transnational research funders who pay reviewers and have
> really robust systems for checking and double checking reviews have
> problems setting scores. It’s basically impossible to do in a completely
> fair way, and even more so in a volunteer system like ours where people
> don’t have much time and reviewers are often assigned to assess papers on
> topics and using methodologles they’re not familiar with. So let’s not make
> the scores part of the feedback to authors.
>
> Not sending scores to authors is an easy fix, and we don’t even have to
> get into all the annual discussions about do 1200 word abstracts even work,
> are humanities perspectives properly evaluated by reviewers, how big should
> the conference be, is it cool that some people present multiple things
> while others are rejected completely etc.
>
> Are there ANY reasons to send scores to authors?
>
> Jill
>
>
> Jill Walker Rettberg
> Professor of Digital Culture
> University of Bergen
>
> Principal Investigator of the ERC Consolidator grant Machine Vision in
> Everyday Life: Playful Interactions with Visual Technologies in Digital
> Art, Games, Narratives and Social Media.
> https://www.uib.no/en/machinevision/
>
>
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at:
> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>


-- 
Professor Tama Leaver
Internet Studies Discipline Lead
School of Media, Creative Arts and Social Inquiry (MCASI)
Faculty of Humanities

Curtin University
GPO Box U1987 Perth WA Australia 6845
Ph: (+61 8) 9266 1258
Email: t.leaver at curtin.edu.au
Web: www.tamaleaver.net
Twitter: @tamaleaver <https://twitter.com/tamaleaver>
CRICOS Provider Code: 00301J (WA)