Industry Specific Intelligent Web Crawler Technology

Student: Matthew Ryan Holmes

Supervisor: Dr. Bob Colomb

Category: Software Engineering Thesis Project

This thesis addresses the design of Internet search and retrieval systems. Specifically, it considers the inclusion of users’ knowledge of a domain into the retrieval process in order to serve industry analysts and decision makers. Inspired by the limitations of current Internet search technologies and the well-understood background of database information storage and query design, this thesis explores the usefulness of customized, client-side Internet information retrieval. To explore the relevance of such search methods, ISIC (Industry-Specific Intelligent Crawler) was designed and the key parts implemented in order to evaluate the feasibility of client-side searching for domain experts. It is concluded that this approach yields a higher prevalence of desired results compared to the commonly used Internet tools, at the cost of increased bandwidth and response time.

Poster Presentation (PDF)

Thesis Document (PDF)