Bibliopedia will perform advanced data-mining and cross-referencing of scholarly literature to create a humanities-centered collaboratory. As a prototype, it will search resources including JSTOR and Library of Congress for metadata about scholarly articles and books that mention the famed medieval travel narrative The Travels of Sir John Mandeville, examine the articles and books for citations, then save the results in a publicly accessible database. Although using works centered around a single text as a test case, Bibliopedia will be designed to work with all humanities scholarship. Bibliopedia will also allow users to create browsable and customizable bibliographies of all the works cited by each article and book. Most importantly, it will use semantic web technology to perform automated textual analysis, data extraction, cross-referencing, and visualizations of the relationships among texts and authors.

Bibliopedia will crawl, aggregate, cross-reference, and display multiple scholarly repositories such as JSTOR, Library of Congress, Project Muse, and Google Books. The project will transform bibliographic data that is currently readable only by humans into machine-readable data via semantic web technology, which will make scholarly resources available for automated analyses, inferences, and visualizations.

SiteCrawler is a website downloading application that allows users to capture entire sites or selected portions of sites like image galleries.

Pattern is a web mining module for Python. It has tools for:

• Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM parser
• Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet
• Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron)
• Network Analysis: graph centrality and visualization.

PDFmyURL converts websites to PDFs. Its very easy and straight forward to use. The Pdf rendered are of good quality. Premium, paid features include watermarks and preference configuration (e.g. paper size, table of contents, disabling/enabling links, etc.)

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP.

DownThemAll is a Firefox plugin that allows users to download all the links or images contained in a webpage.

SiteSucker is OSX and iOS software that can download an entire website, including images and videos.

Heritrix is web crawler used by the Internet Archive, which provides a web-based user interface after initial configuration on a Linux machine. Also used by the Library of Congress, Heritrix captures metadata in the Web ARChive (WARC) format.

HTTrack provides an easy-to-use interface for downloading websites-- including HTML, images, and other files-- or update a copy of a previously-downloaded site.