[Air-l] Mapping the net with crawlers/robots

[DANIELLE WIESE]

Check out Grab-A-Site software from Blue Squirrell.

You can find it through google.

Danielle Wiese
Florida Staet University

>From: "rafel Lucea" <rafel at MIT.EDU>
>Reply-To: air-l at listserv.aoir.org
>To: <air-l-aoir.org at listserv.aoir.org>
>Subject: [Air-l] Mapping the net with crawlers/robots
>Date: Wed, 20 Oct 2004 10:50:27 -0400
>
>Hi all,
>
>I am trying to analyze the relationships between organizations on the
>web. In particular, I want to map the linking behavior of a set of
>organizations subjectively defined.
>
>I have explored a number of software packages (Website Watcher, Sphinx,
>MnoGoSearch, issuecrawler...) but they are either not thought for this
>specific purpose (WW, Issuecrawler) or require coding abilities that are
>beyond my knowledge (Sphinx -OS).
>
>I would be most grateful if someone could indicate me whether there
>exists some web crawler that allows to define
>    - a set of URLs from where to start the crawl
>    - the depth - how many levels one wants to look in a given target
>domain
>    - and number of iterations -how far from the original URL domain one
>wants to go.
>    - and a few filters -limit specific types of pages (pdf for example)
>
>and returns either a map, a table of relationships (some sort of
>adjacency matrix) or both.
>
>Thanks in advance,
>
>Rafel Lucea
>MIT - Sloan School of Management
>
>_______________________________________________
>The Air-l-aoir.org at listserv.aoir.org mailing list
>is provided by the Association of Internet Researchers http://aoir.org
>Subscribe, change options or unsubscribe at: 
>http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
>
>Join the Association of Internet Researchers:
>http://aoir.org/airjoin.html