[Air-L] Request: Stop sending reviewer scores to authors
Jill Walker Rettberg
Jill.Walker.Rettberg at uib.no
Thu Jun 17 21:57:49 PDT 2021
I’ve only used the Easychair system as an administrator - they don’t have this level of detailed scores, it’s more like journal reviewing, with strong accept, accept, weak accept, borderline, weak reject, reject, strong reject as options, plus comments. Reviewers also score their own confidence in their judgement (expert, high, medium, low, none) and the system generates rankings and simple scores (only visible to the admins as a support, not sent to authors) weighted accordingly. Both as a reviewer and as a program chair the reviewers’ own rating of their confidence has been EXTREMELY helpful.
As a program chair dealing with several hundred submissions I found this more than detailed enough, and as a reviewer I find it more precise that the AoIR system. It’s easier to say whether I think an abstract is a weak reject or a strong accept or whatever than you give it a bunch of numerical scores for excellence and coherence - even if I’m not a subject expert.
Jill
________________________________
From: Air-L <air-l-bounces at listserv.aoir.org> on behalf of Tama Leaver <tamaleaver at gmail.com>
Sent: Friday, June 18, 2021 5:38 AM
To: Jean Burgess
Cc: aoir list
Subject: Re: [Air-L] Request: Stop sending reviewer scores to authors
Hi Jean,
That's definitely a good point and something we'll take on board looking at
reviewing for next year. (As you say, I fear the fact that we're not
already might be a technical limitation, but we'll investigate all the
same.)
On that front, it has been lovely that more and more journals are moving in
this direction, too. (Such a relief when we turn out not to be reviewer 2!)
Cheers,
Tama
On Fri, Jun 18, 2021 at 11:21 AM Jean Burgess <jean.burgess at qut.edu.au>
wrote:
> On that, Tama (and I know this isn't the first time someone has raised
> it), one of the most powerful aids to social learning would be to email the
> outcomes to reviewers as well, so that we can see our scores and comments
> alongside those of other reviewers. I understand Conftool may not let you
> do this, but it would be my top request (and like you say, if only our
> major granting bodies would do the same!).
>
> On 17/6/21, 9:14 pm, "Air-L on behalf of Tama Leaver" <
> air-l-bounces at listserv.aoir.org on behalf of tamaleaver at gmail.com> wrote:
>
> Dear Jill,
>
> I'm sorry to hear receiving review scores was a bad experience for you.
>
> Thanks for your suggestion about blinding the scores themselves and
> only
> providing comments, we will take it on board as part of the discussion
> and
> planning for next year's conference.
>
> Since you asked I should add, though, that there are at least two
> functions
> currently achieved by having scores visible.
>
> One is simple transparency: we do use the scores provided as part of
> the
> review process. It isn't the only signal, and this year there was a
> larger
> team so we could spend more time focusing on the qualitative comments
> in
> much more detail, but it is part of the review process and people often
> feel uncomfortable that they can't see part of what is being used to
> judge
> their submission. And of course, in spite of the many hours the
> conference
> committee spends on organising and supervising the review process,
> sometimes a bad review still slips through the cracks. The
> transparency of
> providing the review scores enables our participants to dispute them,
> if
> they feel a score is inappropriate.
>
> (Just from personal experience, I hate that our major grants don't
> show the
> actual scores, just the comments, which clearly sometimes don't align
> well
> at all with the comments people make.)
>
> The second reason is social: seeing our own numbers (whether fair or
> otherwise) is part of the process by which reviewers calibrate what
> fair
> numbers look like. While we do provide guidance on what the numbers
> indicate, sometimes your own past scores are one of the best ways to
> help
> people think about the numbers they provide in their reviews.
>
> Those two points aside, I would also add, there is an internal
> conversation
> already underway about potentially moving away from a 1 to 100 scale to
> something else, and this feedback will definitely be considered in that
> discussion. If you have other suggestions of a productive reviewing
> system
> / review communication system that you’ve experienced elsewhere and
> think
> would be manageable for our organization, being mindful of the scale
> of our
> conference and that we stretch across many disciplines including
> quantitative ones, please do feel free to share your ideas with us.
>
> Many thanks,
> Tama
>
> On Thu, Jun 17, 2021 at 2:19 PM Jill Walker Rettberg <
> Jill.Walker.Rettberg at uib.no> wrote:
>
> > I would like to make a formal request to the AoIR board that for
> future
> > conferences, the organization NOT send out reviewer scores to
> authors.
> >
> > I get why scores are useful for the selection process but it’s hard
> to see
> > how they are helpful for the author, and easy to see how they can be
> > harmful and demotivating. I’m in a secure position and can handle bad
> > reviews, but if I had received the scores I received this year as a
> junior
> > scholar I would have been devastated.
> >
> > The reviewer comments are useful to authors – thank you to all the
> > reviewers for the hard work, and for many constructive comments that
> will
> > be helpful for authors developing their ideas. But there is no
> reason to
> > send the scores to authors.
> >
> > There is also so much research showing the arbitrariness of peer
> review
> > (it’s not random but there is a LOT of subjectivity involved) that
> putting
> > such apparently objective scores out is really misleading. I mean,
> even
> > national and transnational research funders who pay reviewers and
> have
> > really robust systems for checking and double checking reviews have
> > problems setting scores. It’s basically impossible to do in a
> completely
> > fair way, and even more so in a volunteer system like ours where
> people
> > don’t have much time and reviewers are often assigned to assess
> papers on
> > topics and using methodologles they’re not familiar with. So let’s
> not make
> > the scores part of the feedback to authors.
> >
> > Not sending scores to authors is an easy fix, and we don’t even have
> to
> > get into all the annual discussions about do 1200 word abstracts
> even work,
> > are humanities perspectives properly evaluated by reviewers, how big
> should
> > the conference be, is it cool that some people present multiple
> things
> > while others are rejected completely etc.
> >
> > Are there ANY reasons to send scores to authors?
> >
> > Jill
> >
> >
> > Jill Walker Rettberg
> > Professor of Digital Culture
> > University of Bergen
> >
> > Principal Investigator of the ERC Consolidator grant Machine Vision
> in
> > Everyday Life: Playful Interactions with Visual Technologies in
> Digital
> > Art, Games, Narratives and Social Media.
> >
> https://urldefense.com/v3/__https://www.uib.no/en/machinevision/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKnn_D0w9$
> >
> >
> > _______________________________________________
> > The Air-L at listserv.aoir.org mailing list
> > is provided by the Association of Internet Researchers
> https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKsIUTX1n$
> > Subscribe, change options or unsubscribe at:
> >
> https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKrq23j8v$
> >
> > Join the Association of Internet Researchers:
> >
> https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKv_S4L-d$
> >
>
>
> --
> Professor Tama Leaver
> Internet Studies Discipline Lead
> School of Media, Creative Arts and Social Inquiry (MCASI)
> Faculty of Humanities
>
> Curtin University
> GPO Box U1987 Perth WA Australia 6845
> Ph: (+61 8) 9266 1258
> Email: t.leaver at curtin.edu.au
> Web:
> https://urldefense.com/v3/__http://www.tamaleaver.net__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKpRKQrQu$
> Twitter: @tamaleaver <
> https://urldefense.com/v3/__https://twitter.com/tamaleaver__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKiQBM92g$
> >
> CRICOS Provider Code: 00301J (WA)
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers
> https://urldefense.com/v3/__http://aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKsIUTX1n$
> Subscribe, change options or unsubscribe at:
> https://urldefense.com/v3/__http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKrq23j8v$
>
> Join the Association of Internet Researchers:
>
> https://urldefense.com/v3/__http://www.aoir.org/__;!!NVzLfOphnbDXSw!T_Z5pmiAt4IFPzCOvOQKJJH_yMPOnAy6_aG78JH5i-YPX60m-z6gEfHCRo9zKv_S4L-d$
>
>
--
Professor Tama Leaver
Internet Studies Discipline Lead
School of Media, Creative Arts and Social Inquiry (MCASI)
Faculty of Humanities
Curtin University
GPO Box U1987 Perth WA Australia 6845
Ph: (+61 8) 9266 1258
Email: t.leaver at curtin.edu.au
Web: www.tamaleaver.net<http://www.tamaleaver.net>
Twitter: @tamaleaver <https://twitter.com/tamaleaver>
CRICOS Provider Code: 00301J (WA)
_______________________________________________
The Air-L at listserv.aoir.org mailing list
is provided by the Association of Internet Researchers http://aoir.org
Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
Join the Association of Internet Researchers:
http://www.aoir.org/
More information about the Air-L
mailing list