No landslide for the human journalist: An empirical study of computer-generated election news in Finland

Magnus Melin*, Asta Bäck, Caj Södergård, Myriam Munezero, Leo Leppänen

*Corresponding author for this work

    Research output: Contribution to journalArticleScientificpeer-review

    21 Citations (Scopus)

    Abstract

    In an age of struggling news media, automated generation of news via Natural Language Generation (NLG) methods could be of great help, especially in areas where the amount of raw input data is big, and the structure of the data is known in advance. One such news automation system is the Valtteri NLG system, which generates news articles about the Finnish municipal elections of 2017. To evaluate the quality of Valtteri-produced articles, and to identify aspects to improve, n=152 users were asked to evaluate the output of Valtteri. Each evaluator rated six preselected computer-generated articles, four control articles written by journalists, and four computer-generated articles of their own choice. All the articles were evaluated along four dimensions: credibility, liking, quality and representativeness. As expected, the texts written by Valtteri received lower ratings than those written by journalists, but overall the ratings were satisfactory (avg. 2.9 vs. 4.0 for journalists on a 5-point scale). Valtteri´s best rating (3.6) was for credibility. The computer-written articles that the evaluators could freely select got slightly better ratings than the preselected computer-written articles. When looking at the results by demographic groups, males aged 55 or more liked the automatic articles best and females aged 34 or less liked them the least. Evaluators mistook 21% of the computer-written articles as written by humans, and 10% of the human-written articles as computer-written. The share of users making these mistakes grew with the age. Overall, male evaluators made less writer-identification mistakes than female evaluators did.
    Original languageEnglish
    Article number8424161
    Pages (from-to)43356-43367
    JournalIEEE Access
    Volume6
    DOIs
    Publication statusPublished - 1 Aug 2018
    MoE publication typeA1 Journal article-refereed

    Funding

    This work was done as part of the Immersive Automation Project, which is funded by Business Finland, The Media Industry Research Foundation of Finland, The Swedish Cultural Foundation in Finland, the media companies involved as well as the research organizations participating in the project (VTT Technical Research Centre of Finland Ltd., University of Helsinki, Sanoma Media Finland Oy, Alma Media Oyj, Conmio Oy, Keski-Pohjanmaan Kirjapaino Oyj, Streamr, and KSF Media Ab).

    Keywords

    • Artificial intelligence
    • Automated content generation
    • Automated storytelling
    • Natural language processing
    • Robot journalism

    Fingerprint

    Dive into the research topics of 'No landslide for the human journalist: An empirical study of computer-generated election news in Finland'. Together they form a unique fingerprint.

    Cite this