Philip Shilane & Victor Shnayder

 

 

Summary:

Our goal for a memex browser is to archive all of the articles a user views with a web browser and enable efficient searching and retrieval of those articles.  Users will be able to quickly find articles they previously read through queries of their personal archive.

 

The Limits of Memory

As more content has moved to the web, browsing documents through a web browser has become the dominant mechanism for users to view news and scientific documents.  With the proliferation of readily accessible information, users can read dozens of articles a day on a wide range of topics.  While the amount of content a user reads has likely increased because information is so readily available, a human’s ability to remember that information is unchanged. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


(Price per storage capacity is decreasing at an exponential rate.)

 

How often have we been in the situation where we want to find a useful article we read weeks earlier, but we cannot remember the title or authors?  The article might have been a useful reference for a research paper, the itinerary for an upcoming trip, or an interesting movie review we would like to forward to friends.  In the best case, the article is still available on the web and a search engine will be able to help us find the article again.  In the worst case, the article has been removed from the web and there is no way to retrieve information that we previously read.  The most common scenario is that we are able to find the article again, though it may take an extended effort trying the suggestions of a search engine or attempting to retrace the route to the document through related sites. 

 

Even the best web search engines are of limited use for retrieving a previously read article because they were designed to solve a different problem.  General web search engines attempt to use a small number of keywords provided by the user to retrieve a few documents from the billions of pages on the web.  The vast majority of those billions of pages was never viewed by the user and should be eliminated from consideration.  A general search engine is also unable to take advantage of information specific to the user, such as the dates the user read the article. 

 

Features of a Memex Browser

  1. Archive: Every page the user views in a web browser will be stored locally.  We are currently using a proxy that captures http requests.  A new folder is created for every web page that contains the html as well as images referenced from the html.

 

  1. Rewrite web page: Links to images are rewritten to reference local copies of the images.  Links to other pages are currently unchanged, though they could be rewritten to reference a local copy if it is available.  We are currently using wget to download the web pages and perform the rewriting.

 

  1. Parsers:  Besides archiving html, it would be useful to be able to archive and index other types of documents.  As more content is pushed onto the web, it is often in a non-html format because it was originally created for a specialized business purpose, and it was convenient to publish the file in that format.  Parsers will be created for a variety of file types including html, plain text, Adobe PDF, postscript, Microsoft Word, etc.  When a document is viewed by the user, the appropriate parser will analyze the document and update the index.  Standard PDF to text and post script to text translators were used.

 

  1. Document specific viewing:  Since we hope to support a variety of document formats within the memex browser, it is impractical to create new viewers for each document type.  We plan to launch the appropriate application to view the document, Internet Explorer for html documents, Microsoft Word for Word documents, etc. 

 

  1. Being able to annotate entries, add links between entries.  Ideally, this would be done inside the browser, as the user
    is first examining a page. This way, the user can keep track of a graph of related information, and be able to easily recall all that
    info later.  When a user views a page retrieved from the memex, a small window appears that allows the user to make annotations including text comments and links to other pages.

 

  1. Page index:  The index includes information such as the date when a document was first viewed, the most recent date it was viewed, the number of times it was viewed, and a word vector for fast retrieval by keyword.  Standard stop words (e.g. the, and, or, is, and was) are removed from the index. 

 

  1. Search interface:  A powerful user interface will be created for searching based on key words, date originally read, date last read, etc.  Another interesting search capability will be to search for documents within a specific web site.  For example, a user may wish to find all articles about the United Nations that were part of the NY Times site (www.nytimes.com).  The search interface will use the index to perform quick retrieval and present the results to the user in a grid format that allows the user to rank the results by date, number of times read, file format, etc.  The user can select whether to search the page index and/or the image index.  From the search interface, the user will be able to select an item and view it with a standard web browser.

 

Prototype

We have built prototype applications that implement some of the above features.  The first component is a local Internet proxy that records the http requests into a file.  A script periodically checks the file for newly viewed pages and uses the wget program to download the page and its referenced images.  The page is then rewritten to reference the local copies of the images.  Standard translators were used to convert PDF and postscript files into text that was added to the index.

 

We used MySQL to store the web page information including date, URL, text, page type, as well as meta information added by the user including notes and links to other interesting sites.

 

Search Features

My prototype, shown below, allows the user to specify a variety of search criteria.  The user can enter a set of search terms that must appear in the web pages.  Page types can be specified including html, pdf, postscript, and Word documents.  The base URL can also be specified, either by selecting from the list of most frequent URL’s, or by entering the URL separately.  The date when the page was first viewed can also be selected, either entering the number of months or days in the past to search, or by entering a range of dates within which to search.

(User is searching for html pages containing the term “computer” within www.google.com between December 1, 2002 and January 24, 2003)

 

 

(Searching for all html pages with the term Linux within the site www.linux.org viewed within the last five months.)

 

 

Adding comments

Beyond storing documents viewed by a user, Vannevar Bush’s original description of the memex would also store meta-data about the pages entered by a user.  This meta-data could link together a series of documents that are interrelated or that shows the user’s progression of thought.  I implemented features to store comments and links with a document.  When a user views a document in the memex browser, a window appears that allows the user to enter notes and links to other documents, which is then stored in the database.  When a user views that document in the future, the meta-data previously entered will appear again as a reminder of his/her previous work.

 

(A window for storing meta-data about the document appears, allowing the user to store notes and links to other documents.)

 

 

(The user is viewing a document about the situation with Iraq, and has created links to other related documents.)

 

 

 

Memex Statistics

From November 2002 through January 2003, Victor stored the pages he viewed through his web browser in the memex database.  The database included 5,784 documents totaling 720 Mb of storage space.  Continuing at this pace, the memex database would index approximately 23,000 documents and 2.8 Gb of data each year. 

 

 

 

References

    • Using Memex to archive and mine community Web browsing experience. Soumen Chakrabarti, Sandeep Srivastava, Mallela Subramanyam, Mitul Tiwari.  Indian Institute of Technology Bombay. Online at http://www9.org/w9cdrom/98/98.html.

 

 

 

 

 

    • Section 8.12, "Putting it All Together: The Google Cluster of PCs," Computer Architecture, A Quantitative Approach, Third Edition. John L. Hennessey and David A. Patterson, pages 855-862, 2003.