Validating Untrained Human Annotations Using Extreme Learning Machines

Thomas Forss; Leonardo Espinosa-Leal; Anton Akusok; Amaury Lendasse; Kaj-Mikael Björk

doi:10.1007/978-3-030-58989-9_10

Validating Untrained Human Annotations Using Extreme Learning Machines

Thomas Forss, Leonardo Espinosa-Leal^*, Anton Akusok, Amaury Lendasse, Kaj-Mikael Björk

^*Corresponding author for this work

Not published at VTT

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

Abstract

We present a process for validating and improving annotations made by untrained humans using a two-step machine learning algorithm. The initial validation algorithm is trained on a high quality annotated subset of the data that the untrained humans are asked to annotate. We continue by using the machine learning algorithm to predict other samples that are also annotated by the humans and test several approaches for joining the algorithmic annotations with the human annotations, with the intention of improving the performance beyond using either approach individually. We show that combining human annotations with the algorithmic predictions can improve the accuracy of the annotations.

Original language	English
Title of host publication	Proceedings of ELM2019
Editors	Jiuwen Cao, Chi Man Vong, Yoan Miche, Amaury Lendasse
Place of Publication	Cham
Publisher	Springer
Pages	89-98
ISBN (Electronic)	978-3-030-58989-9
ISBN (Print)	978-3-030-58988-2, 978-3-030-59049-9
DOIs	https://doi.org/10.1007/978-3-030-58989-9_10
Publication status	Published - 12 Sept 2020
MoE publication type	A4 Article in a conference publication
Event	2019 International Conference on Extreme Learning Machine (ELM 2019) - Yangzhou, China Duration: 14 Dec 2019 → 16 Dec 2019

Publication series

Series	Proceedings in Adaptation, Learning and Optimization
Volume	14
ISSN	2363-6084

Conference

Conference	2019 International Conference on Extreme Learning Machine (ELM 2019)
Country/Territory	China
City	Yangzhou
Period	14/12/19 → 16/12/19

Access to Document

10.1007/978-3-030-58989-9_10

Cite this

@inproceedings{fcd765bd96f64a55a58eeaa67e010ff2,

title = "Validating Untrained Human Annotations Using Extreme Learning Machines",

abstract = "We present a process for validating and improving annotations made by untrained humans using a two-step machine learning algorithm. The initial validation algorithm is trained on a high quality annotated subset of the data that the untrained humans are asked to annotate. We continue by using the machine learning algorithm to predict other samples that are also annotated by the humans and test several approaches for joining the algorithmic annotations with the human annotations, with the intention of improving the performance beyond using either approach individually. We show that combining human annotations with the algorithmic predictions can improve the accuracy of the annotations.",

author = "Thomas Forss and Leonardo Espinosa-Leal and Anton Akusok and Amaury Lendasse and Kaj-Mikael Bj{\"o}rk",

year = "2020",

month = sep,

day = "12",

doi = "10.1007/978-3-030-58989-9_10",

language = "English",

isbn = "978-3-030-58988-2",

series = "Proceedings in Adaptation, Learning and Optimization",

publisher = "Springer",

pages = "89--98",

editor = "Jiuwen Cao and Vong, {Chi Man} and Yoan Miche and Amaury Lendasse",

booktitle = "Proceedings of ELM2019",

address = "Germany",

note = "2019 International Conference on Extreme Learning Machine (ELM 2019) ; Conference date: 14-12-2019 Through 16-12-2019",

}

Forss, T, Espinosa-Leal, L, Akusok, A, Lendasse, A & Björk, K-M 2020, Validating Untrained Human Annotations Using Extreme Learning Machines. in J Cao, CM Vong, Y Miche & A Lendasse (eds), Proceedings of ELM2019. Springer, Cham, Proceedings in Adaptation, Learning and Optimization, vol. 14, pp. 89-98, 2019 International Conference on Extreme Learning Machine (ELM 2019), Yangzhou, China, 14/12/19. https://doi.org/10.1007/978-3-030-58989-9_10

Validating Untrained Human Annotations Using Extreme Learning Machines. / Forss, Thomas; Espinosa-Leal, Leonardo; Akusok, Anton et al.
Proceedings of ELM2019. ed. / Jiuwen Cao; Chi Man Vong; Yoan Miche; Amaury Lendasse. Cham: Springer, 2020. p. 89-98 (Proceedings in Adaptation, Learning and Optimization, Vol. 14).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Validating Untrained Human Annotations Using Extreme Learning Machines

AU - Forss, Thomas

AU - Espinosa-Leal, Leonardo

AU - Akusok, Anton

AU - Lendasse, Amaury

AU - Björk, Kaj-Mikael

PY - 2020/9/12

Y1 - 2020/9/12

N2 - We present a process for validating and improving annotations made by untrained humans using a two-step machine learning algorithm. The initial validation algorithm is trained on a high quality annotated subset of the data that the untrained humans are asked to annotate. We continue by using the machine learning algorithm to predict other samples that are also annotated by the humans and test several approaches for joining the algorithmic annotations with the human annotations, with the intention of improving the performance beyond using either approach individually. We show that combining human annotations with the algorithmic predictions can improve the accuracy of the annotations.

AB - We present a process for validating and improving annotations made by untrained humans using a two-step machine learning algorithm. The initial validation algorithm is trained on a high quality annotated subset of the data that the untrained humans are asked to annotate. We continue by using the machine learning algorithm to predict other samples that are also annotated by the humans and test several approaches for joining the algorithmic annotations with the human annotations, with the intention of improving the performance beyond using either approach individually. We show that combining human annotations with the algorithmic predictions can improve the accuracy of the annotations.

U2 - 10.1007/978-3-030-58989-9_10

DO - 10.1007/978-3-030-58989-9_10

M3 - Conference article in proceedings

SN - 978-3-030-58988-2

SN - 978-3-030-59049-9

T3 - Proceedings in Adaptation, Learning and Optimization

SP - 89

EP - 98

BT - Proceedings of ELM2019

A2 - Cao, Jiuwen

A2 - Vong, Chi Man

A2 - Miche, Yoan

A2 - Lendasse, Amaury

PB - Springer

CY - Cham

T2 - 2019 International Conference on Extreme Learning Machine (ELM 2019)

Y2 - 14 December 2019 through 16 December 2019

ER -

Validating Untrained Human Annotations Using Extreme Learning Machines

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this