Web Scrapers
A list of tools for scraping text from various places online.
<p>Google News Scraper is a free, web-based tool for batch querying news.google.com. Users can specify search terms, news sources, date range, location, language, version of Google (ex. .com, .ca, .co.uk, etc.), and what facets to include in the output. The tool returns the results as a table suitable for further analysis, and can also include thumbnail images for each search item.</p>
<p>The Google Blogsearch Scraper is a free, web-based tool for batch querying Google Blog Search. Users can submit a list of URLs to search, and a set of keywords to apply to them. For each query, the tool supplies a tag cloud and an HTML table, plus a text file that may be downloaded for further analysis.</p>
This free web-based tool allows you to scrape the comments from a YouTube video and save them in either a JSON or CSV format. The project is open source, and its code is available on github.
A very simple/specialized piece of Java code that takes a csv file produced by "Youtube Comment Scraper" - http://ytcomments.klostermann.ca/ and collapses the replies down into the same categories that basic comments are under. This makes it easier to compare all comments to each other.
NOTE: The name of the csv file to be parsed has to be written into the code, assigned to the variable "inFile".
<p>Twitter Capture and Analysis Toolset (DMI-TCAT) is a free, open source tool for capturing and analyzing tweets. The tool's web interface is currently closed to researchers outside the University of Amsterdam's Media Studies department, however, the source code is available for download and installation via Github.</p>
WIScker is a tool for generating corpi from Wikipedia articles. It takes all of the various versions and edits of an article over time and creates a comparative corpus out of them. The corpus can then be analyzed using a built-in button to access voyant tools, or can be exported in a text file in various different formats.
Other list by Gregory Whistance-Smith
Sentiment Analysis