Skip to main content

Topic-Sensitive Hidden-Web Crawling.

A constantly growing amount of high-quality information is stored in pages coming from the Hidden Web. Such pages are accessible only through a query interface that a Hidden-Web site provides and may span a variety of topics.

In order to provide centralized access to the Hidden Web, previous works have focused on query generation techniques that aim at downloading all content of a given Hidden Web site with the minimum cost. In certain settings however, we are interested in downloading only a specific part of such a site. For example, in a news database, a user may be interested in retrieving only sports articles but no politics. In this case, we need to make the best use of our resources in downloading only the portion of the Hidden Web site that we are interested in.

In this paper, we study how we can build a topically-focused Hidden Web crawler that can autonomously extract topic-specific pages from the Hidden Web by searching only the subset that is related to the corresponding category. To this end, we present query generation techniques that take into account the topic that we are interested in. We propose a number of different crawling policies and we experimentally evaluate them with data from two popular sites.


Panagiotis Liakos, Alexandros Ntoulas, "Topic-Sensitive Hidden-Web Crawling. ", Web Information Systems Engineering - WISE 2012 - 13th International Conference, 2012
Published at
Web Information Systems Engineering - WISE 2012 - 13th International Conference
Related research area
No related research area
Related Organizations
No related organizations