[Air-L] urlte.am to back up URL shorteners

[gus andrews]

So here's a project which should be tremendously useful to Internet
researchers, just announced by Jason Scott, a "rogue archivist activist":
URLTE.AM. The project will apparently be making backups of every URL coded
for by a URL shortener, thereby keeping track when links go dead and expire.
Between this and archive.org, a lot less Internet history should be lost.

>From the description at urlte.am:

"Welcome to the URLTeam website. The URLTeam is the ArchiveTeam subcommittee
on URL shorteners. We believe that they pose a serious threat to the
internet's integrity. If one of them dies, gets hacked or sells out,
millions of links will stop working. Thus we preemptively release backups,
because URL shorteners are too busy to make backups themselves.

Releases
Every 6 months or so we release a torrent of all backed up files. When a new
torrent is released, you can simply delete the old torrent and download the
new files to the same location. Your BitTorrent client will figure out which
files have changed and will redownload them, even if you weren't finished
yet on the previous torrent.

The latest torrent was released on May 31th, 2011: urlteam.torrent - List of
files - Readme

The next release is planned around December 2011.

Data format
All data files in the torrent are simple text files compressed using
LZMA2/xz. The text file format is very simple: Each line contains one
mapping in the following format: Shortcode, pipe (Ascii 0x7C), long URL,
line feed (Ascii 0x10). The file is sorted by shortcodes using the following
order:

Shorter shortcodes come before longer ones
Decimal digits
Lowercase letters
Uppercase letters
ASCII value
Depending on the URL shortener there might be multiple long URLs for one
shortcode.

There are some tools for working with the files on GitHub.

Q&A
Can you do a backup of shortener XY please?
Maybe. Some shorteners are very fast at banning scrapers which makes it
impossible to do a backup in an efficient way. Contact us and we will look
into it.

What about 301Works.org? They help URL shorteners with backups.
Unfortunately they rely on the cooperation of URL shorteners, and many of
the biggest URL shorteners refuse to cooperate. Furthermore, they don't plan
on releasing any data files to the public. We do however greatly value their
work and when selecting which URL shorteners we will scrape next we
concentrate especially on those that don't cooperate with 301Works.

Since March 2011 we are actively uploading data from non-cooperating
shorteners to the 301Works archive. While those files are not available for
download (they contain the same data as our torrents anyway) you can watch
our progress here.

I like what you do. Can I help?
Sure thing. We can always use people who help with scraping. Or programmers.
We could also use a fast server with lots of space for storing the data and
seeding the torrents. Anyhow, if you want to help, please contact us.

What's with all those weird directory and file names in the torrent?
It's hard to organize that stuff into files so that each individual file is
only a few hundred megabytes in size. Because we want to accomodate people
with inferior operating systems we also need to assume a case insensitive
file system."

Gillian "Gus" Andrews
Postdoctoral Fellow, Google Academic Research Grant
Anthropology in Education
Teachers College, Columbia University
http://gandre.ws