Increasing payments in crowdsourcing: Don't look a gift horse in the mouth!

Martín Varela, Toni Mäki, Lea Skorin-Kapov, Tobias Hoßfeld

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientific

Abstract

A commonly cited maxim states that "you get what you pay for", implying that there is a strong correlation between the price paid for something, and its quality. In this paper, we examine whether this often-cited wisdom applies to using crowdsourcing for conducting subjective QoE experiments, and if so, how. As part of a large-scale user study designed to assess Web QoE, we conducted two crowdsourced campaigns to collect user ratings and study the influence of certain website design parameters related to typography and color on the overall visual appeal of the site. While the test content was exactly the same across both campaigns, the second campaign was set up to pay participants three times the reward of the first one. The goal was to analyze the impact of payment on a number of parameters, including the ratio of reliable users and obtained MOS values. With respect to QoE modeling, we found that while payment levels influenced absolute MOS values, there was no significant impact on the actual model.
Original languageEnglish
Title of host publicationProceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013
EditorsRaimund Schatz, Tobias Hoßfeld
Pages24-29
Publication statusPublished - 2013
MoE publication typeNot Eligible
Event4th International Workshop on Perceptual Quality of Systems, PQS 2013 - Vienna, Austria
Duration: 2 Sep 20134 Sep 2013

Conference

Conference4th International Workshop on Perceptual Quality of Systems, PQS 2013
Abbreviated titlePQS 2013
CountryAustria
CityVienna
Period2/09/134/09/13

    Fingerprint

Keywords

  • crowdsourcing
  • quality assessment
  • incentives
  • Quality of Experience

Cite this

Varela, M., Mäki, T., Skorin-Kapov, L., & Hoßfeld, T. (2013). Increasing payments in crowdsourcing: Don't look a gift horse in the mouth! In R. Schatz, & T. Hoßfeld (Eds.), Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013 (pp. 24-29) http://pqs.qu.tu-berlin.de/wp-content/uploads/2016/08/PQS2013_Proceedings.pdf