Eye Tracker in the Wild

Studying the delta between what is said and measured in a crowdsourcing experiment

Pierre Leberton, Isabelle Hupont, Toni Mäki, Evangelos Skodras, Matthias Hirth

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

3 Citations (Scopus)

Abstract

Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non-intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users' side are a webcam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.
Original languageEnglish
Title of host publicationCrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia
PublisherAssociation for Computing Machinery ACM
Pages3-8
ISBN (Print)978-1-4503-3746-5
DOIs
Publication statusPublished - 2015
MoE publication typeA4 Article in a conference publication
Event4th International Workshop on Crowdsourcing for Multimedia, CrowdMM 2015 - Brisbane, Australia
Duration: 30 Oct 201530 Oct 2015
Conference number: 4

Conference

Conference4th International Workshop on Crowdsourcing for Multimedia, CrowdMM 2015
Abbreviated titleCrowdMM 2015
CountryAustralia
CityBrisbane
Period30/10/1530/10/15

Fingerprint

Web browsers
Experiments
Calibration
Hardware
Monitoring

Keywords

  • cognitive science
  • human-centered computing
  • human computer interaction
  • HCI
  • artificial intelligence
  • computing methodologies

Cite this

Leberton, P., Hupont, I., Mäki, T., Skodras, E., & Hirth, M. (2015). Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment. In CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia (pp. 3-8). Association for Computing Machinery ACM. https://doi.org/10.1145/2810188.2810192
Leberton, Pierre ; Hupont, Isabelle ; Mäki, Toni ; Skodras, Evangelos ; Hirth, Matthias. / Eye Tracker in the Wild : Studying the delta between what is said and measured in a crowdsourcing experiment. CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia. Association for Computing Machinery ACM, 2015. pp. 3-8
@inproceedings{b63dd3f11a104b91a56f3b0876559d8a,
title = "Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment",
abstract = "Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non-intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users' side are a webcam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.",
keywords = "cognitive science, human-centered computing, human computer interaction, HCI, artificial intelligence, computing methodologies",
author = "Pierre Leberton and Isabelle Hupont and Toni M{\"a}ki and Evangelos Skodras and Matthias Hirth",
year = "2015",
doi = "10.1145/2810188.2810192",
language = "English",
isbn = "978-1-4503-3746-5",
pages = "3--8",
booktitle = "CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia",
publisher = "Association for Computing Machinery ACM",
address = "United States",

}

Leberton, P, Hupont, I, Mäki, T, Skodras, E & Hirth, M 2015, Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment. in CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia. Association for Computing Machinery ACM, pp. 3-8, 4th International Workshop on Crowdsourcing for Multimedia, CrowdMM 2015, Brisbane, Australia, 30/10/15. https://doi.org/10.1145/2810188.2810192

Eye Tracker in the Wild : Studying the delta between what is said and measured in a crowdsourcing experiment. / Leberton, Pierre; Hupont, Isabelle; Mäki, Toni; Skodras, Evangelos; Hirth, Matthias.

CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia. Association for Computing Machinery ACM, 2015. p. 3-8.

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

TY - GEN

T1 - Eye Tracker in the Wild

T2 - Studying the delta between what is said and measured in a crowdsourcing experiment

AU - Leberton, Pierre

AU - Hupont, Isabelle

AU - Mäki, Toni

AU - Skodras, Evangelos

AU - Hirth, Matthias

PY - 2015

Y1 - 2015

N2 - Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non-intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users' side are a webcam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.

AB - Self-reported metrics collected in crowdsourcing experiments do not always match the actual user behaviour. Therefore in the laboratory studies the visual attention, the capability of humans to selectively process the visual information with which they are confronted, is traditionally measured by means of eye trackers. Visual attention has not been typically considered in crowdsourcing environments, mainly because of the requirements of specific hardware and challenging gaze calibration. This paper proposes the use of a non-intrusive eye tracking crowdsourcing framework, where the only technical requirements from the users' side are a webcam and a HTML5 compatible web browser, to study the differences between what a participant implicitly and explicitly does during a crowdsourcing experiment. To demonstrate the feasibility of this approach, an exemplary crowdsourcing campaign was launched to collect and compare both eye tracking data and self-reported metrics from the users. Participants performed a movie selection task, where they were asked about the main reasons motivating them to choose a particular movie. Results demonstrate the added value of monitoring gaze in crowdsourcing contexts: consciously or not, users behave differently than what they report through questionnaires.

KW - cognitive science

KW - human-centered computing

KW - human computer interaction

KW - HCI

KW - artificial intelligence

KW - computing methodologies

U2 - 10.1145/2810188.2810192

DO - 10.1145/2810188.2810192

M3 - Conference article in proceedings

SN - 978-1-4503-3746-5

SP - 3

EP - 8

BT - CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia

PB - Association for Computing Machinery ACM

ER -

Leberton P, Hupont I, Mäki T, Skodras E, Hirth M. Eye Tracker in the Wild: Studying the delta between what is said and measured in a crowdsourcing experiment. In CrowdMM '15 Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia. Association for Computing Machinery ACM. 2015. p. 3-8 https://doi.org/10.1145/2810188.2810192