[Air-L] News sites are locking out the Internet Archive to stop AI crawling. Is the ‘open web’ closing?

[Sharath]

I actively use the Way Back Machine (an Internet Archive project), to
obtain full pdf articles behind a paywall, be it The Guardian, The Atlantic
or even the New Yorker. These are usually a  pre-edited snapshot, which is
sufficient for me and in fact much more interesting because sometimes you
get to read sentences that the web editor chose to remove.

Also the equivalent of Napster and Magnet Torrent of the late 90s in
today’s world of YouTube shorts, Instagram /FB reels and news videos , are
open source tools such as yt-dlp (formerly YouTube-dl).
YouTube-dl GitHub was closed down in the pandemic when some political
videos made it across platforms despite the original source removing them.
But now yt-dlp is back and I can even download instagram/fb reels and news
broadcasts embedded in websites that are designed to explicitly prevent
downloading.

Similarly I am sure a mirror of the Internet archive and way back machine
will be accessible either using the Tor browser or an alternate DnS.

Also extending Lili’s observation, when transparency/open access is
introduced into a non neutral system, it will always used to
disproportionately benefit those in power.

 I suspect newsrooms are more concerned about competitors reading their
pre-published snapshots and producing similar content overnight, rather
than ‘pirate’ consumers.

Regards

On Saturday, February 7, 2026, Lilly Irani via Air-L <
air-l at listserv.aoir.org> wrote:

> It reminds me of work on the commons and biopolitics / pharmaceuticals from
> the early 2000s. Anthropologists Cori Hayden and Kimberly Christen showed
> separately two days ago the way "openness" can be exploited by well
> equipped companies and governments. Hayden wrote about pharmaceutical
> companies exploiting biocommons in drug development in violation of
> indegenous rights and protocols. Christen wrote about Aboriginal people in
> Australia preventing their photos from being accessed outside of ethical
> protocols in museums, a violation of openness but an ethical choice.
>
> I'm still an open access fan for publishing my own scholarly work *for
> people* but I do not believe Anthropic should train their AIs without
> asking.
>
> In India, Anil Gupta developed a sharing license / patent for openness in
> farming where non-profit and small businesses could share intellectual
> property but large corporations have to pay the patent owners. This is not
> an unsolved problem.
>
> On Fri, Feb 6, 2026 at 11:44 PM Caroline James-Garrod via Air-L <
> air-l at listserv.aoir.org> wrote:
>
> > Thanks Tai. Very important to know.
> >
> > Best wishes to you,
> >
> > Caroline
> >
> > Get Outlook for Android<
> > https://urldefense.com/v3/__https://aka.ms/AAb9ysg__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINABfQVaOn$
> > >
> > ________________________________
> > From: Air-L <air-l-bounces at listserv.aoir.org> on behalf of Tai Neilson
> > via Air-L <air-l at listserv.aoir.org>
> > Sent: Thursday, February 5, 2026 7:38:18 PM
> > To: air-l at listserv.aoir.org <air-l at listserv.aoir.org>
> > Subject: [Air-L] News sites are locking out the Internet Archive to stop
> > AI crawling. Is the ‘open web’ closing?
> >
> > Dear AoIR community,
> > Apologies for cross‑posting, but I wanted to share a recent article of
> > mine in The Conversation: News sites are locking out the Internet Archive
> > to stop AI crawling. Is the ‘open web’ closing?<
> > https://urldefense.com/v3/__https://theconversation.com/
> news-sites-are-locking-out-the-internet-archive-to-stop-
> ai-crawling-is-the-open-web-closing-274968__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINABW53PCb$
> > >
> > It addresses the further enclosure of the internet, commodification of
> > publishers' archives as AI training data, and the resulting challenges
> for
> > nonprofits and internet research. I hope you find it useful.
> > Cheers,
> > Tai
> >
> >
> > Tai Neilson, PhD
> > Senior Lecturer in Media
> > Major Coordinator of Journalism and Nonfiction Writing
> > MACQUARIE UNIVERSITY NSW 2109
> > _______________________________________________
> > The Air-L at listserv.aoir.org mailing list
> > is provided by the Association of Internet Researchers
> > https://urldefense.com/v3/__http://aoir.org__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINAKJmRHBl$
> > Subscribe, change options or unsubscribe at:
> > https://urldefense.com/v3/__http://listserv.aoir.org/
> listinfo.cgi/air-l-aoir.org__;!!Mih3wA!C_CoGqIMq2jeSbwHccYEuDlW9KRZODJy
> QLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWXiKuINANEAmqcD$
> >
> > Join the Association of Internet Researchers:
> >
> > https://urldefense.com/v3/__http://www.aoir.org__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINADpnLsQo$
> >
> > _______________________________________________
> > The Air-L at listserv.aoir.org mailing list
> > is provided by the Association of Internet Researchers
> > https://urldefense.com/v3/__http://aoir.org__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINAKJmRHBl$
> > Subscribe, change options or unsubscribe at:
> > https://urldefense.com/v3/__http://listserv.aoir.org/
> listinfo.cgi/air-l-aoir.org__;!!Mih3wA!C_CoGqIMq2jeSbwHccYEuDlW9KRZODJy
> QLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWXiKuINANEAmqcD$
> >
> > Join the Association of Internet Researchers:
> >
> > https://urldefense.com/v3/__http://www.aoir.org/__;!!Mih3wA!C_
> CoGqIMq2jeSbwHccYEuDlW9KRZODJyQLen-8GV4EBqNbkQhJ9L7Q8WQI98ZiOZpWX
> iKuINAIONwlQ4$
> >
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/
> listinfo.cgi/air-l-aoir.org
>
> Join the Association of Internet Researchers:
> http://www.aoir.org/
>


-- 
*SHARATH CHANDRA RAMAKRISHNAN*
*Perceptual Futures Laboratory *
*Assistant Professor, School of Art & Design*
*Affiliate Assistant Professor, School of Information Sciences*
University of Illinois Urbana-Champaign
scram at illinois.edu

<http://illinois.edu/>