2012

MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture

Anand Kulkarni, Philipp Gutheim, Prayag Narula, David Rolnitzky, Tapan Parikh, and Bjoern Hartmann, “MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture”, IEEE Internet Computing, September-October 2012

Abstract

Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks. A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that experienced members of the crowd prevent wrong answers. Last, social interaction techniques give the best workers the ability and incentives to manage, teach and supervise other members of the crowd, as well as to clarify tasks. This process allows the crowd to collaboratively learn how to solve unfamiliar tasks.

Last updated: July 19, 2019