Keyword Extraction for Webpage Clusters

The volume of unstructured information presented on the Internet is constantly increasing, together with the total amount of websites and their contents. To process this vast amount of information it is important to distinguish different clusters of related webpages. Such clusters are used, for example, for template induction, keyword extraction, and recommendation algorithms. A variety of applications (such as semantic analysis systems, crawlers and search engines) utilize clustering algorithms to recognize thematically connected webpages. The clustering is performed based on sets of webpage features, for instance, hyperlinks between webpages, search logs or DOM tree structures. In this article we present an approach for keyword extraction, which utilizes two different clustering algorithms. The first algorithm is based on the analysis of the textual information, the second one– on the statistical information (obtained from the Google Analytics API). We evaluate our approach with three different corpuses of webpages.Also, we compare our keyword extraction approach with a non-clustered baseline and show a significant improvement in the relevance of extracted keywords.