[Air-L] Request: Stop sending reviewer scores to authors

Thu Jun 17 20:37:35 PDT 2021

Hi Jean,

That's definitely a good point and something we'll take on board looking at
reviewing for next year.  (As you say, I fear the fact that we're not
already might be a technical limitation, but we'll investigate all the
same.)

On that front, it has been lovely that more and more journals are moving in
this direction, too. (Such a relief when we turn out not to be reviewer 2!)

Cheers,
Tama

On Fri, Jun 18, 2021 at 11:21 AM Jean Burgess <jean.burgess at qut.edu.au>
wrote:

> On that, Tama (and I know this isn't the first time someone has raised
> it), one of the most powerful aids to social learning would be to email the
> outcomes to reviewers as well, so that we can see our scores and comments
> alongside those of other reviewers. I understand Conftool may not let you
> do this, but it would be my top request (and like you say, if only our
> major granting bodies would do the same!).
>
> On 17/6/21, 9:14 pm, "Air-L on behalf of Tama Leaver" <
> air-l-bounces at listserv.aoir.org on behalf of tamaleaver at gmail.com> wrote:
>
>     Dear Jill,
>
>     I'm sorry to hear receiving review scores was a bad experience for you.
>
>     Thanks for your suggestion about blinding the scores themselves and
> only
>     providing comments, we will take it on board as part of the discussion
> and
>     planning for next year's conference.
>
>     Since you asked I should add, though, that there are at least two
> functions
>     currently achieved by having scores visible.
>
>     One is simple transparency: we do use the scores provided as part of
> the
>     review process. It isn't the only signal, and this year there was a
> larger
>     team so we could spend more time focusing on the qualitative comments
> in
>     much more detail, but it is part of the review process and people often
>     feel uncomfortable that they can't see part of what is being used to
> judge
>     their submission. And of course, in spite of the many hours the
> conference
>     committee spends on organising and supervising the review process,
>     sometimes a bad review still slips through the cracks. The
> transparency of
>     providing the review scores enables our participants to dispute them,
> if
>     they feel a score is inappropriate.
>
>     (Just from personal experience, I hate that our major grants don't
> show the
>     actual scores, just the comments, which clearly sometimes don't align
> well
>     at all with the comments people make.)
>
>     The second reason is social: seeing our own numbers (whether fair or
>     otherwise) is part of the process by which reviewers calibrate what
> fair
>     numbers look like. While we do provide guidance on what the numbers
>     indicate, sometimes your own past scores are one of the best ways to
> help
>     people think about the numbers they provide in their reviews.
>
>     Those two points aside, I would also add, there is an internal
> conversation
>     already underway about potentially moving away from a 1 to 100 scale to
>     something else, and this feedback will definitely be considered in that
>     discussion.  If you have other suggestions of a productive reviewing
> system
>     / review communication system that you’ve experienced elsewhere and
> think
>     would be manageable for our organization, being mindful of the scale
> of our
>     conference and that we stretch across many disciplines including
>     quantitative ones, please do feel free to share your ideas with us.
>
>     Many thanks,
>     Tama
>
>     On Thu, Jun 17, 2021 at 2:19 PM Jill Walker Rettberg <
>     Jill.Walker.Rettberg at uib.no> wrote:
>
>     > I would like to make a formal request to the AoIR board that for
> future
>     > conferences, the organization NOT send out reviewer scores to
> authors.
>     >
>     > I get why scores are useful for the selection process but it’s hard
> to see
>     > how they are helpful for the author, and easy to see how they can be
>     > harmful and demotivating. I’m in a secure position and can handle bad
>     > reviews, but if I had received the scores I received this year as a
> junior
>     > scholar I would have been devastated.
>     >
>     > The reviewer comments are useful to authors – thank you to all the
>     > reviewers for the hard work, and for many constructive comments that
> will
>     > be helpful for authors developing their ideas. But there is no
> reason to
>     > send the scores to authors.
>     >
>     > There is also so much research showing the arbitrariness of peer
> review
>     > (it’s not random but there is a LOT of subjectivity involved) that
> putting
>     > such apparently objective scores out is really misleading. I mean,
> even
>     > national and transnational research funders who pay reviewers and
> have
>     > really robust systems for checking and double checking reviews have
>     > problems setting scores. It’s basically impossible to do in a
> completely
>     > fair way, and even more so in a volunteer system like ours where
> people
>     > don’t have much time and reviewers are often assigned to assess
> papers on
>     > topics and using methodologles they’re not familiar with. So let’s
> not make
>     > the scores part of the feedback to authors.
>     >
>     > Not sending scores to authors is an easy fix, and we don’t even have
> to
>     > get into all the annual discussions about do 1200 word abstracts
> even work,
>     > are humanities perspectives properly evaluated by reviewers, how big
> should
>     > the conference be, is it cool that some people present multiple
> things
>     > while others are rejected completely etc.
>     >
>     > Are there ANY reasons to send scores to authors?
>     >
>     > Jill
>     >
>     >
>     > Jill Walker Rettberg
>     > Professor of Digital Culture
>     > University of Bergen
>     >
>     > Principal Investigator of the ERC Consolidator grant Machine Vision
> in
>     > Everyday Life: Playful Interactions with Visual Technologies in
> Digital
>     > Art, Games, Narratives and Social Media.
>     >
> https://urldefense.com/v3/__https://www.uib.no/en/machinevision/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKnn_D0w9$
>     >
>     >
>     > _______________________________________________
>     > The Air-L at listserv.aoir.org mailing list
>     > is provided by the Association of Internet Researchers
> https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKsIUTX1n$
>     > Subscribe, change options or unsubscribe at:
>     >
> https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKrq23j8v$
>     >
>     > Join the Association of Internet Researchers:
>     >
> https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKv_S4L-d$
>     >
>
>
>     --
>     Professor Tama Leaver
>     Internet Studies Discipline Lead
>     School of Media, Creative Arts and Social Inquiry (MCASI)
>     Faculty of Humanities
>
>     Curtin University
>     GPO Box U1987 Perth WA Australia 6845
>     Ph: (+61 8) 9266 1258
>     Email: t.leaver at curtin.edu.au
>     Web:
> https://urldefense.com/v3/__http://www.tamaleaver.net__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKpRKQrQu$
>     Twitter: @tamaleaver <
> https://urldefense.com/v3/__https://twitter.com/tamaleaver__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKiQBM92g$
> >
>     CRICOS Provider Code: 00301J (WA)
>     _______________________________________________
>     The Air-L at listserv.aoir.org mailing list
>     is provided by the Association of Internet Researchers
> https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKsIUTX1n$
>     Subscribe, change options or unsubscribe at:
> https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKrq23j8v$
>
>     Join the Association of Internet Researchers:
>
> https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKv_S4L-d$
>
>

-- 
Professor Tama Leaver
Internet Studies Discipline Lead
School of Media, Creative Arts and Social Inquiry (MCASI)
Faculty of Humanities

Curtin University
GPO Box U1987 Perth WA Australia 6845
Ph: (+61 8) 9266 1258
Email: t.leaver at curtin.edu.au
Web: www.tamaleaver.net
Twitter: @tamaleaver <https://twitter.com/tamaleaver>
CRICOS Provider Code: 00301J (WA)