Istituto di Scienza e Tecnologie dell'Informazione     
Berardi G., Esuli A., Sebastiani F. A utility-theoretic ranking method for semi-automated text classification. In: SIGIR'12 - The 35th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (Portland, Oregon, USA, 12-16 August 2012). Proceedings, pp. 961 - 970. ACM, 2012.
In Semi-Automated Text Classification (SATC) an automatic classifier Phi labels a set of unlabelled documents D, following which a human annotator inspects (and corrects when appropriate) the labels attributed by Phi to a subset D' of D, with the aim of improving the overall quality of the labelling. An automated system can support this process by ranking the automatically labelled documents in a way that maximizes the expected increase in effectiveness that derives from inspecting D'. An obvious strategy is to rank D so that the documents that Phi has classified with the lowest confidence are top-ranked. In this work we show that this strategy is suboptimal. We develop a new utility-theoretic ranking method based on the notion of inspection gain, defined as the improvement in classification effectiveness that would derive by inspecting and correcting a given automatically labelled document. We also propose a new effectiveness measure for SATC-oriented ranking methods, based on the expected reduction in classification error brought about by partially inspecting a list generated by a given ranking method. We report the results of experiments showing that, with respect to the baseline method above, and according to the proposed measure, our ranking method can achieve substantially higher expected reductions in classification error.
Subject Text classification
Supervised learning
Semi-automated text classification
Cost-sensitive learning
I.5.0 Pattern recognition. General

Icona documento 1) Download Document PDF

Icona documento Open access Icona documento Restricted Icona documento Private


Per ulteriori informazioni, contattare: Librarian http://puma.isti.cnr.it

Valid HTML 4.0 Transitional