[Air-L] Academic replacements for TwapperKeeper.com?

[Cornelius Puschmann]

*Note:* I've also blogged this (in case links in the post don't work) and
will list all alternatives suggested to me in that blog post:
http://blog.ynada.com/616

Dear all,

A few days ago, the people behind Twitter archival site
TwapperKeeper.com<http://twapperkeeper.com/> announced
that they will be discontinuing the export feature of the service on March
20, 2011<http://twapperkeeper.wordpress.com/2011/02/22/removal-of-export-and-download-api-capabilities/>.
Apparently the feature is in violation of Twitter’s terms of service, at
least in the form it’s currently implemented in TwapperKeeper.

Unfortunately this cuts off a number of academics who are investigating
communication on Twitter for scientific purposes from a convenient data
source. While it’s fairly easy to get data directly via the Twitter
API<http://apiwiki.twitter.com/> (which
is what TwapperKeeper was doing), I know many people who want to concentrate
on the data itself, rather than running their own servers to scrape Twitter
on a regular basis. What’s more is that Twitter’s attitude is worrisome:
many of us have tried to get an exemption from API rate limits in the past,
to no avail. Twitter doesn’t give researchers privileged access to their
data, and now they’re crippling TwapperKeeper on top of that.

Bottom line: what will we use after March 20? Ideally, a replacement would
provide the following:

   - the hashtag/search query functionality of TwapperKeeper,
   - the export functionality of TwapperKeeper,
   - exclusive use for academic purposes (on the grounds that this might
   keep Twitter from shutting it down),
   - stability and reliability,
   - long-term viability.

The last point is important, because I don’t think it will be difficult to
set up a server somewhere to suit the needs of a few people, but a
larger-scale solution seems more sensible in the long run. Maybe
JISC<http://www.jisc.ac.uk/> can
do something like that, based
onyourTwapperKeeper<http://code.google.com/p/yourtwapperkeeper/>
 (which they supported<http://twapperkeeper.wordpress.com/2010/04/16/jisc-funded-developments-to-twapper-keeper/>)?
Or one of the big institutes (OII, Berkman)? Either way it would be nice to
find an alternative that doesn’t give those of us with devs and major IT
support behind them a huge edge over the rest…

Thanks in advance for your comments,

Cornelius

---

Dr. Cornelius Puschmann
Department of English Language and Linguistics
Heinrich-Heine-University Düsseldorf, Germany
Junior Researchers Group "Science and the Internet"
http://nfgwin.uni-duesseldorf.de