Istituto di Scienza e Tecnologie dell'Informazione     
Caropreso M. F., Matwin S., Sebastiani F. Statistical Phrases in Automated Text Categorization. Technical report, 2000.
In this work we investigate the usefulness of n-grams for document indexing in text categorization (TC). We call n-gram a set tk of n word stems, and we say that tk occurs in a document dj when a sequence of words appears in dj that, after stop word removal and stemming, consists exactly of the n stems in tk in some order. Previous researches have investigated the use of n-grams (or some variant of them) in the context of specific learning algorithms, and thus have not obtained general answers on their usefulness for TC. In this work we investigate the usefulness of n-grams in TC independently of any specific learning algorithm. We do so by applying feature selection to the pool of all α-grams (α ≤ n), and checking how many n-grams score high enough to be selected in the top σ α-grams. We report the results of our experiments, using several feature selection functions and varying values of σ, performed on the Reuters-21578 standard TC benchmark. We also report results of making actual use of the selected n-grams in the context of a linear classifier induced by means of the Rocchio method.
Subject Machine learning
Text categorisation
Text classification
H.3.3 Information filtering
H.3.3 Performance evaluation (efficiency and effectiveness)
I.2.3 Induction

Icona documento 1) Download Document PDF

Icona documento Open access Icona documento Restricted Icona documento Private


Per ulteriori informazioni, contattare: Librarian http://puma.isti.cnr.it

Valid HTML 4.0 Transitional