PUMA
Istituto di Scienza e Tecnologie dell'Informazione     
Baccianella S., Esuli A., Sebastiani F. Evaluation measures for ordinal regression. In: ISDA'09 - 9th IEEE International Conference on Intelligent Systems Design and Applications (Pisa, IT, November 30 - December 2 2009). Proceedings, pp. 283 - 287. IEEE Press, 2009.
 
 
Abstract
(English)
Ordinal regression (OR -- also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.
Subject Ordinal regression
Ordinal classification
Evaluation measures
I.2.6 Learning (K.3.2)
I.5.2 Design Methodology. Classifier design and evaluation


Icona documento 1) Download Document PDF


Icona documento Open access Icona documento Restricted Icona documento Private

 


Per ulteriori informazioni, contattare: Librarian http://puma.isti.cnr.it

Valid HTML 4.0 Transitional