An apple-to-apple comparison of Learning-to-rank algorithms in terms of Normalized Discounted Cumulative Gain

R. Busa-Fekete G. Szarvas T. Élteto B. Kégl 1, 2
1 TAO - Machine Learning and Optimisation
Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, CNRS - Centre National de la Recherche Scientifique : UMR8623, LRI - Laboratoire de Recherche en Informatique
2 Appstat
LAL - Laboratoire de l'Accélérateur Linéaire, LRI - Laboratoire de Recherche en Informatique
Abstract : The Normalized Discounted Cumulative Gain (NDCG) is a widely used evaluation metric for learning-to-rank (LTR) systems. NDCG is designed for ranking tasks with more than one relevance levels. There are many freely available, open source tools for computing the NDCG score for a ranked result list. Even though the definition of NDCG is unambiguous, the various tools can produce different scores for ranked lists with certain properties, deteriorating the empirical tests in many published papers and thereby making the comparison of empirical results published in different studies difficult to compare. In this study, first, we identify the major differences between the various publicly available NDCG evaluation tools. Second, based on a set of comparative experiments using a common benchmark dataset in LTR research and 6 different LTR algorithms, we demonstrate how these differences affect the overall performance of different algorithms and the final scores that are used to compare different systems.
Type de document :
Communication dans un congrès
De Raedt, L., Bessiere, C., Dubois, D., Doherty, P., Frasconi, P., Heintz, F., Lucas, P. 20th European Conference on Artificial Intelligence (ECAI 2012) : Preference Learning: Problems and Applications in AI Workshop, Aug 2012, Montpellier, France. Ios Press, 242, 2012


http://hal.in2p3.fr/in2p3-00726760
Contributeur : Sabine Starita <>
Soumis le : vendredi 31 août 2012 - 11:27:55
Dernière modification le : mercredi 20 juillet 2016 - 09:44:52
Document(s) archivé(s) le : samedi 1 décembre 2012 - 03:30:57

Fichier

07-busa-fekete.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : in2p3-00726760, version 1

Collections

Citation

R. Busa-Fekete, G. Szarvas, T. Élteto, B. Kégl. An apple-to-apple comparison of Learning-to-rank algorithms in terms of Normalized Discounted Cumulative Gain. De Raedt, L., Bessiere, C., Dubois, D., Doherty, P., Frasconi, P., Heintz, F., Lucas, P. 20th European Conference on Artificial Intelligence (ECAI 2012) : Preference Learning: Problems and Applications in AI Workshop, Aug 2012, Montpellier, France. Ios Press, 242, 2012. <in2p3-00726760>

Exporter

Partager

Métriques

Consultations de
la notice

373

Téléchargements du document

363