Improving Document Representation for Story Link Detection by Modeling Term Topicality

DOI Web Site 4 References Open Access
  • Shah Chirag
    National Institute of Informatics the University of North Carolina at Chapel Hill
  • Eguchi Koji
    National Institute of Informatics Kobe University

Description

Several information organization, access, and filtering systems can benefit from different kind of document representations than those used in traditional Information Retrieval (IR). Topic Detection and Tracking (TDT) is an example of such a domain. In this paper we demonstrate that traditional methods for term weighing do not capture topical information and this leads to inadequate representation of documents for TDT applications. We present various hypotheses regarding the factors that can help in improving the document representation for Story Link Detection (SLD) — a core task of TDT. These hypotheses are tested using various TDT collections. From our experiments and analysis we found that in order to obtain a faithful representation of documents in TDT domain, we not only need to capture a term's importance in traditional IR sense, but also evaluate its topical behavior. Along with defining this behavior, we propose a novel measure that captures a term's importance at the collection level as well as its discriminating power for topics. This new measure leads to a much better document representation as reflected by the significant improvements in the results.

Journal

References(4)*help

See more

Keywords

Details 詳細情報について

Report a problem

Back to top