Increasing payments in crowdsourcing

Don't look a gift horse in the mouth!

Martín Varela, Toni Mäki, Lea Skorin-Kapov, Tobias Hoßfeld

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientific

Abstract

A commonly cited maxim states that "you get what you pay for", implying that there is a strong correlation between the price paid for something, and its quality. In this paper, we examine whether this often-cited wisdom applies to using crowdsourcing for conducting subjective QoE experiments, and if so, how. As part of a large-scale user study designed to assess Web QoE, we conducted two crowdsourced campaigns to collect user ratings and study the influence of certain website design parameters related to typography and color on the overall visual appeal of the site. While the test content was exactly the same across both campaigns, the second campaign was set up to pay participants three times the reward of the first one. The goal was to analyze the impact of payment on a number of parameters, including the ratio of reliable users and obtained MOS values. With respect to QoE modeling, we found that while payment levels influenced absolute MOS values, there was no significant impact on the actual model.
Original languageEnglish
Title of host publicationProceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013
EditorsRaimund Schatz, Tobias Hoßfeld
Pages24-29
Publication statusPublished - 2013
MoE publication typeNot Eligible
Event4th International Workshop on Perceptual Quality of Systems, PQS 2013 - Vienna, Austria
Duration: 2 Sep 20134 Sep 2013

Conference

Conference4th International Workshop on Perceptual Quality of Systems, PQS 2013
Abbreviated titlePQS 2013
CountryAustria
CityVienna
Period2/09/134/09/13

Fingerprint

Color
Experiments
Web Design

Keywords

  • crowdsourcing
  • quality assessment
  • incentives
  • Quality of Experience

Cite this

Varela, M., Mäki, T., Skorin-Kapov, L., & Hoßfeld, T. (2013). Increasing payments in crowdsourcing: Don't look a gift horse in the mouth! In R. Schatz, & T. Hoßfeld (Eds.), Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013 (pp. 24-29)
Varela, Martín ; Mäki, Toni ; Skorin-Kapov, Lea ; Hoßfeld, Tobias. / Increasing payments in crowdsourcing : Don't look a gift horse in the mouth!. Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013. editor / Raimund Schatz ; Tobias Hoßfeld. 2013. pp. 24-29
@inproceedings{d05e833b2dd949188b146f9a36c630b5,
title = "Increasing payments in crowdsourcing: Don't look a gift horse in the mouth!",
abstract = "A commonly cited maxim states that {"}you get what you pay for{"}, implying that there is a strong correlation between the price paid for something, and its quality. In this paper, we examine whether this often-cited wisdom applies to using crowdsourcing for conducting subjective QoE experiments, and if so, how. As part of a large-scale user study designed to assess Web QoE, we conducted two crowdsourced campaigns to collect user ratings and study the influence of certain website design parameters related to typography and color on the overall visual appeal of the site. While the test content was exactly the same across both campaigns, the second campaign was set up to pay participants three times the reward of the first one. The goal was to analyze the impact of payment on a number of parameters, including the ratio of reliable users and obtained MOS values. With respect to QoE modeling, we found that while payment levels influenced absolute MOS values, there was no significant impact on the actual model.",
keywords = "crowdsourcing, quality assessment, incentives, Quality of Experience",
author = "Mart{\'i}n Varela and Toni M{\"a}ki and Lea Skorin-Kapov and Tobias Ho{\ss}feld",
note = "Project code: 75001",
year = "2013",
language = "English",
pages = "24--29",
editor = "Raimund Schatz and Tobias Ho{\ss}feld",
booktitle = "Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013",

}

Varela, M, Mäki, T, Skorin-Kapov, L & Hoßfeld, T 2013, Increasing payments in crowdsourcing: Don't look a gift horse in the mouth! in R Schatz & T Hoßfeld (eds), Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013. pp. 24-29, 4th International Workshop on Perceptual Quality of Systems, PQS 2013, Vienna, Austria, 2/09/13.

Increasing payments in crowdsourcing : Don't look a gift horse in the mouth! / Varela, Martín; Mäki, Toni; Skorin-Kapov, Lea; Hoßfeld, Tobias.

Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013. ed. / Raimund Schatz; Tobias Hoßfeld. 2013. p. 24-29.

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientific

TY - GEN

T1 - Increasing payments in crowdsourcing

T2 - Don't look a gift horse in the mouth!

AU - Varela, Martín

AU - Mäki, Toni

AU - Skorin-Kapov, Lea

AU - Hoßfeld, Tobias

N1 - Project code: 75001

PY - 2013

Y1 - 2013

N2 - A commonly cited maxim states that "you get what you pay for", implying that there is a strong correlation between the price paid for something, and its quality. In this paper, we examine whether this often-cited wisdom applies to using crowdsourcing for conducting subjective QoE experiments, and if so, how. As part of a large-scale user study designed to assess Web QoE, we conducted two crowdsourced campaigns to collect user ratings and study the influence of certain website design parameters related to typography and color on the overall visual appeal of the site. While the test content was exactly the same across both campaigns, the second campaign was set up to pay participants three times the reward of the first one. The goal was to analyze the impact of payment on a number of parameters, including the ratio of reliable users and obtained MOS values. With respect to QoE modeling, we found that while payment levels influenced absolute MOS values, there was no significant impact on the actual model.

AB - A commonly cited maxim states that "you get what you pay for", implying that there is a strong correlation between the price paid for something, and its quality. In this paper, we examine whether this often-cited wisdom applies to using crowdsourcing for conducting subjective QoE experiments, and if so, how. As part of a large-scale user study designed to assess Web QoE, we conducted two crowdsourced campaigns to collect user ratings and study the influence of certain website design parameters related to typography and color on the overall visual appeal of the site. While the test content was exactly the same across both campaigns, the second campaign was set up to pay participants three times the reward of the first one. The goal was to analyze the impact of payment on a number of parameters, including the ratio of reliable users and obtained MOS values. With respect to QoE modeling, we found that while payment levels influenced absolute MOS values, there was no significant impact on the actual model.

KW - crowdsourcing

KW - quality assessment

KW - incentives

KW - Quality of Experience

M3 - Conference article in proceedings

SP - 24

EP - 29

BT - Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013

A2 - Schatz, Raimund

A2 - Hoßfeld, Tobias

ER -

Varela M, Mäki T, Skorin-Kapov L, Hoßfeld T. Increasing payments in crowdsourcing: Don't look a gift horse in the mouth! In Schatz R, Hoßfeld T, editors, Proceedings of the 4th International Workshop on Perceptual Quality of Systems PQS 2013. 2013. p. 24-29