MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture

Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks. A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that experienced members of the crowd prevent wrong answers. Last, social interaction techniques give the best workers the ability and incentives to manage, teach and supervise other members of the crowd, as well as to clarify tasks. This process allows the crowd to collaboratively learn how to solve unfamiliar tasks.