[Air-L] ‘The Worst Internet-Research Ethics Violation I Have Ever Seen’

Mon May 5 23:43:05 PDT 2025

Thanks for sharing this. There is a history of methodological discussions
around the arbitrary boundary between covert and unethical research. What
I've read on the topic has been f2f ethnographic, but I think could/should
be applied to cases like this to expand the discussion to digital/social
media platforms as well as to content-analytic and other social
media-relevant methods.

https://journals.sagepub.com/doi/abs/10.1177/0038038510387195

https://doi.org/10.4135/9781473920835

-p.

On Sat, May 3, 2025 at 6:59 PM Joly MacFie via Air-L <
air-l at listserv.aoir.org> wrote:

> [posted in full for discussion purposes]
>
>
> https://www.theatlantic.com/technology/archive/2025/05/reddit-ai-persuasion-experiment-ethics/682676/
>
> By Tom Bartlett
>
> When Reddit rebranded itself as “the heart of the internet” a couple of
> years ago, the slogan was meant to evoke the site’s organic character. In
> an age of social media dominated by algorithms, Reddit took pride in being
> curated by a community that expressed its feelings in the form of upvotes
> and downvotes—in other words, being shaped by actual people.
>
> So earlier this week, when members of a popular subreddit learned that
> their community had been infiltrated by undercover researchers posting
> AI-written comments and passing them off as human thoughts, the Redditors
> were predictably incensed. They called the experiment “violating,”
> “shameful,” “infuriating,” and “very disturbing.” As the backlash
> intensified, the researchers went silent, refusing to reveal their identity
> or answer questions about their methodology. The university that employs
> them has announced that it’s investigating. Meanwhile, Reddit’s chief legal
> officer, Ben Lee, wrote that the company intends to “ensure that the
> researchers are held accountable for their misdeeds.”
>
> Joining the chorus of disapproval were fellow internet researchers, who
> condemned what they saw as a plainly unethical experiment. Amy Bruckman, a
> professor at the Georgia Institute of Technology who has studied online
> communities for more than two decades, told me the Reddit fiasco is “the
> worst internet-research ethics violation I have ever seen, no contest.”
> What’s more, she and others worry that the uproar could undermine the work
> of scholars who are using more conventional methods to study a crucial
> problem: how AI influences the way humans think and relate to one another.
>
> The researchers, based at the University of Zurich, wanted to find out
> whether AI-generated responses could change people’s views. So they headed
> to the aptly named subreddit r/changemyview, in which users debate
> important societal issues, along with plenty of trivial topics, and award
> points to posts that talk them out of their original position. Over the
> course of four months, the researchers posted more than 1,000 AI-generated
> comments on pitbulls (is aggression the fault of the breed or the owner?),
> the housing crisis (is living with your parents the solution?), DEI
> programs (were they destined to fail?). The AI commenters argued that
> browsing Reddit is a waste of time and that the “controlled demolition”
> 9/11 conspiracy theory has some merit. And as they offered their
> computer-generated opinions, they also shared their backstories. One
> claimed to be a trauma counselor; another described himself as a victim of
> statutory rape.
>
> In one sense, the AI comments appear to have been rather effective. When
> researchers asked the AI to personalize its arguments to a Redditor’s
> biographical details, including gender, age, and political leanings
> (inferred, courtesy of another AI model, through the Redditor’s post
> history), a surprising number of minds indeed appear to have been changed.
> Those personalized AI arguments received, on average, far higher scores in
> the subreddit’s point system than nearly all human commenters, according to
> preliminary findings that the researchers shared with Reddit moderators and
> later made private. (This analysis, of course, assumes that no one else in
> the subreddit was using AI to hone their arguments.)
>
> The researchers had a tougher time convincing Redditors that their covert
> study was justified. After they had finished the experiment, they contacted
> the subreddit’s moderators, revealed their identity, and requested to
> “debrief” the subreddit—that is, to announce to members that for months,
> they had been unwitting subjects in a scientific experiment. “They were
> rather surprised that we had such a negative reaction to the experiment,”
> says one moderator, who asked to be identified by his username,
> LucidLeviathan, to protect his privacy. According to LucidLeviathan, the
> moderators requested that the researchers not publish such tainted work,
> and that they issue an apology. The researchers refused. After more than a
> month of back-and-forth, the moderators revealed what they had learned
> about the experiment (minus the researchers’ names) to the rest of the
> subreddit, making clear their disapproval.
>
> When the moderators sent a complaint to the University of Zurich, the
> university noted in its response that the “project yields important
> insights, and the risks (e.g. trauma etc.) are minimal,” according to an
> excerpt posted by moderators. In a statement to me, a university
> spokesperson said that the ethics board had received notice of the study
> last month, advised the researchers to comply with the subreddit’s rules,
> and “intends to adopt a stricter review process in the future.” Meanwhile,
> the researchers defended their approach in a Reddit comment, arguing that
> “none of the comments advocate for harmful positions” and that each
> AI-generated comment was reviewed by a human team member before being
> posted. (I sent an email to an anonymized address for the researchers,
> posted by Reddit moderators, and received a reply that directed my
> inquiries to the university.)
>
> Perhaps the most telling aspect of the Zurich researchers’ defense was
> that, as they saw it, deception was integral to the study. The University
> of Zurich’s ethics board—which can offer researchers advice but, according
> to the university, lacks the power to reject studies that fall short of its
> standards—told the researchers before they began posting that “the
> participants should be informed as much as possible,” according to the
> university statement I received. But the researchers seem to believe that
> doing so would have ruined the experiment. “To ethically test LLMs’
> persuasive power in realistic scenarios, an unaware setting was necessary,”
> because it more realistically mimics how people would respond to
> unidentified bad actors in real-world settings, the researchers wrote in
> one of their Reddit comments.
>
> How humans are likely to respond in such a scenario is an urgent issue and
> a worthy subject of academic research. In their preliminary results, the
> researchers concluded that AI arguments can be “highly persuasive in
> real-world contexts, surpassing all previously known benchmarks of human
> persuasiveness.” (Because the researchers finally agreed this week not to
> publish a paper about the experiment, the accuracy of that verdict will
> probably never be fully assessed, which is its own sort of shame.) The
> prospect of having your mind changed by something that doesn’t have one is
> deeply unsettling. That persuasive superpower could also be employed for
> nefarious ends.
>
> Still, scientists don’t have to flout the norms of experimenting on human
> subjects in order to evaluate the threat. “The general finding that AI can
> be on the upper end of human persuasiveness—more persuasive than most
> humans—jibes with what laboratory experiments have found,” Christian
> Tarsney, a senior research fellow at the University of Texas at Austin,
> told me. In one recent laboratory experiment, participants who believed in
> conspiracy theories voluntarily chatted with an AI; after three exchanges,
> about a quarter of them lost faith in their previous beliefs. Another found
> that ChatGPT produced more persuasive disinformation than humans, and that
> participants who were asked to distinguish between real posts and those
> written by AI could not effectively do so.
>
> Giovanni Spitale, the lead author of that study, also happens to be a
> scholar at the University of Zurich, and has been in touch with one of the
> researchers behind the Reddit AI experiment, who asked him not to reveal
> their identity. “We are receiving dozens of death threats,” the researcher
> wrote to him, in a message Spitale shared with me. “Please keep the secret
> for the safety of my family.”
>
> One likely reason the backlash has been so strong is because, on a platform
> as close-knit as Reddit, betrayal cuts deep. “One of the pillars of that
> community is mutual trust,” Spitale told me; it’s part of the reason he
> opposes experimenting on Redditors without their knowledge. Several
> scholars I spoke with about this latest ethical quandary compared
> it—unfavorably—to Facebook’s infamous emotional-contagion study. For one
> week in 2012, Facebook altered users’ News Feed to see if viewing more or
> less positive content changed their posting habits. (It did, a little bit.)
> Casey Fiesler, an associate professor at the University of Colorado at
> Boulder who studies ethics and online communities, told me that the
> emotional-contagion study pales in comparison with what the Zurich
> researchers did. “People were upset about that but not in the way that this
> Reddit community is upset,” she told me. “This felt a lot more personal.”
>
> The reaction probably also has to do with the unnerving notion that ChatGPT
> knows what buttons to push in our minds. It’s one thing to be fooled by
> some human Facebook researchers with dubious ethical standards, and another
> entirely to be duped by a cosplaying chatbot. I read through dozens of the
> AI comments, and although they weren’t all brilliant, most of them seemed
> reasonable and genuine enough. They made a lot of good points, and I found
> myself nodding along more than once. As the Zurich researchers warn,
> without more robust detection tools, AI bots might “seamlessly blend into
> online communities”—that is, assuming they haven’t already.
>
>
>
> --
> --------------------------------------
> Joly MacFie  +12185659365
> --------------------------------------
> -
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at:
> http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>