COS 435
Resources for Projects

Please help keep this list useful!  Suggest free, stable, preferably open-source software and free, open data sets.  It is best if you can give a recommendation from personal experience, but we will take other suggestions subject to further exploration.  Thanks!


Data Sets
Here are some data sets of possible value.  Some have been used in past COS435 projects.

IMPORTANT NOTE:  Other members of the faculty have data sets that they are willing to share.  If you need something - either a specific data set or a specific kind of data - ask, and we'll see if it is available in the department.

• UCI Machine Learning Archive - University of California at Irving data sets, primarily for data mining tasks, but also useful for other information analysis/search tasks.  211 Data Sets as of February 2012.  Some example data sets:
• NSF Research Awards Abstracts 1990-2003
• OpinRank Review Dataset including car reviews for model years 2007-2009 and hotel reviews for 10 cities.
  

• LETOR data set:  From Microsoft.  The site says "a package of benchmark data sets for research on LEarning TO Rank. This dataset contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines, for the OHSUMED data collection and the '.gov' data collection." 

• Stanford Large Network Dataset Collection:  Networks of all kinds:  social networks, Web graphs, Internet networks to name a few.

• Amazon Web Services (AWS) Public Data Sets:  " a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications" Examples: Sloan Digital Sky Survey,  a 5 billion Web page crawl (60TB!).  Many others.
  

• 4 universities data set:  from CMU.   CS department Web pages from various universities, hand-classified into 7 categories.

• The Open American National Corpus:  11,242 documents in English

• Wikimedia Downloads, including Wikipedia

• Collections provided on Web site for Search Engines:  Information Retrieval in Practice  a textbook by Croft, Metzler and Strohman.

Software
Here are some free (as far as I know) software tools of possible value.  Some have been used in past COS435 projects.

• open source web crawler WebSphinx  
  
• Java toolkit for indexing and search:  Lucene
• Probabalistic Latent Semantic Analysis (pLSA) Matlab implementation by Jakob Verbeek.
• Latent Dirichlet allocation (LDA) in C by David Blei.
  
• Correlated topic model (CTM) in C  by David Blei.
• Twitter API
• YouTube data API
• Natural Language Toolkit (NLTK):  From the site: "Open source Python modules, linguistic data and documentation for research and development in natural language processing, supporting dozens of NLP tasks, with distributions for Windows, Mac OSX and Linux."  Includes access to WordNet.
• Stemming algorithm evaluation program:  developed by Rob Hooper, Lancaster University
• Lancaster  stemming algorithm designed by Paice and Husk of Lancaster University
• XAMPP:   From the site "an easy to install Apache distribution containing MySQL, PHP and Perl"
• The Lemur Toolkit for Language Modeling and Information Retrieval
• Galago toolkit for experimenting with text search - used in Search Engines:  Information Retrieval in Practice by Croft, Metzler and Strohman
  

---

lists last updated Fri Feb 17 14:03:58 EST 2012
Copyright  2008-2012  Andrea S. LaPaugh