Istituto di Scienza e Tecnologie dell'Informazione     
Caropreso M. F., Matwin S., Sebastiani F. A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization. In: 'Text Databases and Document Management: Theory and Practice'. Chapter IV, 78-102. By Amita Goyal Chin. Hershey: Idea Group Publishing, 2001. 2001.
In this work we investigate the usefulness of {em $n$-grams} for document indexing in text categorization (TC). We call $n$-gram a set $g_k$ of $n$ word stems, and we say that $g_k$ occurs in a document $d_j$ when a sequence of words appears in $d_j$ that, after stop word removal and stemming, consists exactly of the $n$ stems in $g_k$, in some order. Previous researches have investigated the use of $n$-grams (or some variant of them) in the context of specific learning algorithms, and thus have not obtained general answers on their usefulness for TC. In this work we investigate the usefulness of $n$-grams in TC independently of any specific learning algorithm. We do so by applying feature selection to the pool of all $k$-grams ($kleq n$), and checking how many $n$-grams score high enough to be selected in the top $sigma$ $k$-grams. We report the results of our experiments, using various feature selection measures and varying values of $sigma$, performed on the {sc Reuters-21578} standard TC benchmark. We also report results of making actual use of the selected $n$-grams in the context of a linear classifier induced by means of the Rocchio method.
Subject document indexing in text categorization
H.3.3 Information Search and Retrieval

Icona documento 1) Download Document PDF

Icona documento Open access Icona documento Restricted Icona documento Private


Per ulteriori informazioni, contattare: Librarian http://puma.isti.cnr.it

Valid HTML 4.0 Transitional