Crawl.py Crawl.py is a threaded web crawler that crawls a specified domain (or collection of domains) and collects metadata about each page it visits. This crawler can be used to analyze the performance of a web server, analyze relationships between pages, or traverse a site and scrape pages. Data collected includes: - url & parsed url (scheme/netloc/path/etc...) - page load time in milliseconds - page size in bytes - link addresses on the page - number of links on the page - number of links within the page domain - number of links targeting external domains Links on each page are recorded in the 'url_canonical' table. Visits to each link are recorded in the 'visit_metadata' table. Relationships between a visited link and all the links in the response are recorded in the 'page_rel' table. Planned functionality: - addition of an optional 'scrape page' function - global toggle to allow already-visited URLs to be requeued. - usage guidelines and documentation Known issues: - URL encoding is sometimes incorrect, which may cause duplication issues and does result in errors opening requests