Dynamic Estimation of Rater Reliability in Regression Tasks using Multi-Armed Bandit Techniques

Alexey Tarasov, Dublin Institute of Technology
Sarah Jane Delany, Dublin Institute of Technology
Brian Mac Namee, Dublin Institute of Technology

Document Type Conference Paper

Presented at the Workshop on Machine Learning in Human Computation and Crowdsourcing, in conjunction with ICML 2012


In this paper we show that MAB techniques are suit- able for performing the task of the dynamic estimation of rater reliability. We focus on crowdsourcing scenar- ios where the ratings are numerical values typically in a scale, in contrast to other research in the area which concentrates on binary ratings.