Performance-based usability evaluation of a safety information and alarm system

Leena Norros (Corresponding Author), Maaria Nuutinen

Research output: Contribution to journalArticleScientificpeer-review

28 Citations (Scopus)

Abstract

Evaluation of the appropriateness of information technical systems for complex professional usage in safety-critical contexts poses significant methodical and practical challenges. In this study, the usability of a Safety Information and Alarm Panel (SIAP) in a nuclear power plant control room was tested. An integrated validation concept was used that included a new approach to measuring system and operator performance in complex work environments. The tested system was designed to aid the operators in severe disturbance and emergency situations. It had already been implemented at a nuclear power plant. The study was conducted in a full-scope training simulator. The results verified that an acceptable level of performance could be achieved when using the SIAP. When the operators’ practices were analysed by a habit-centred analysis, it was discovered that the effects of the SIAP differed between crews and between test scenarios. Thus, the SIAP tended to promote coherence of practices but reduce situatively attentive action. In diffuse task contexts the tool failed to support the shift supervisor's control of the overall process situation, his awareness of the crew's work load and his ability to update the crew's awareness of the process. The operators reported that the system supported their process control activity and reduced stress in the situation, but the shift supervisors and operators also noticed some possible negative effects of the tool. These subjective evaluations corresponded to the effects observed in practice. The results revealed the complexity of the implementation of new tools into professional practice. It was proposed that a validation project should focus on the trajectory of development of the entire distributed cognitive system instead of comprehending validation studies as tests of the effects of information systems on a pre-defined process output. Formative evaluation criteria are needed in projecting distributed cognitive systems.
Original languageEnglish
Pages (from-to)328 - 361
Number of pages34
JournalInternational Journal of Human-Computer Studies
Volume63
Issue number3
DOIs
Publication statusPublished - 2005
MoE publication typeA1 Journal article-refereed

Fingerprint

Alarm systems
Information systems
Cognitive systems
evaluation
performance
Supervisory personnel
nuclear power plant
Nuclear power plants
control process
Process control
work environment
Simulators
habits
Trajectories
information system
scenario
ability

Keywords

  • validation method
  • information presentation
  • control room
  • operators
  • practice
  • nuclear power plants
  • SIAP
  • nuclear reactor safety
  • safety
  • alarm systems

Cite this

@article{f2b388f0d2554e53a333b757e0b45864,
title = "Performance-based usability evaluation of a safety information and alarm system",
abstract = "Evaluation of the appropriateness of information technical systems for complex professional usage in safety-critical contexts poses significant methodical and practical challenges. In this study, the usability of a Safety Information and Alarm Panel (SIAP) in a nuclear power plant control room was tested. An integrated validation concept was used that included a new approach to measuring system and operator performance in complex work environments. The tested system was designed to aid the operators in severe disturbance and emergency situations. It had already been implemented at a nuclear power plant. The study was conducted in a full-scope training simulator. The results verified that an acceptable level of performance could be achieved when using the SIAP. When the operators’ practices were analysed by a habit-centred analysis, it was discovered that the effects of the SIAP differed between crews and between test scenarios. Thus, the SIAP tended to promote coherence of practices but reduce situatively attentive action. In diffuse task contexts the tool failed to support the shift supervisor's control of the overall process situation, his awareness of the crew's work load and his ability to update the crew's awareness of the process. The operators reported that the system supported their process control activity and reduced stress in the situation, but the shift supervisors and operators also noticed some possible negative effects of the tool. These subjective evaluations corresponded to the effects observed in practice. The results revealed the complexity of the implementation of new tools into professional practice. It was proposed that a validation project should focus on the trajectory of development of the entire distributed cognitive system instead of comprehending validation studies as tests of the effects of information systems on a pre-defined process output. Formative evaluation criteria are needed in projecting distributed cognitive systems.",
keywords = "validation method, information presentation, control room, operators, practice, nuclear power plants, SIAP, nuclear reactor safety, safety, alarm systems",
author = "Leena Norros and Maaria Nuutinen",
year = "2005",
doi = "10.1016/j.ijhcs.2005.03.004",
language = "English",
volume = "63",
pages = "328 -- 361",
journal = "International Journal of Human-Computer Studies",
issn = "1071-5819",
publisher = "Academic Press",
number = "3",

}

Performance-based usability evaluation of a safety information and alarm system. / Norros, Leena (Corresponding Author); Nuutinen, Maaria.

In: International Journal of Human-Computer Studies, Vol. 63, No. 3, 2005, p. 328 - 361.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Performance-based usability evaluation of a safety information and alarm system

AU - Norros, Leena

AU - Nuutinen, Maaria

PY - 2005

Y1 - 2005

N2 - Evaluation of the appropriateness of information technical systems for complex professional usage in safety-critical contexts poses significant methodical and practical challenges. In this study, the usability of a Safety Information and Alarm Panel (SIAP) in a nuclear power plant control room was tested. An integrated validation concept was used that included a new approach to measuring system and operator performance in complex work environments. The tested system was designed to aid the operators in severe disturbance and emergency situations. It had already been implemented at a nuclear power plant. The study was conducted in a full-scope training simulator. The results verified that an acceptable level of performance could be achieved when using the SIAP. When the operators’ practices were analysed by a habit-centred analysis, it was discovered that the effects of the SIAP differed between crews and between test scenarios. Thus, the SIAP tended to promote coherence of practices but reduce situatively attentive action. In diffuse task contexts the tool failed to support the shift supervisor's control of the overall process situation, his awareness of the crew's work load and his ability to update the crew's awareness of the process. The operators reported that the system supported their process control activity and reduced stress in the situation, but the shift supervisors and operators also noticed some possible negative effects of the tool. These subjective evaluations corresponded to the effects observed in practice. The results revealed the complexity of the implementation of new tools into professional practice. It was proposed that a validation project should focus on the trajectory of development of the entire distributed cognitive system instead of comprehending validation studies as tests of the effects of information systems on a pre-defined process output. Formative evaluation criteria are needed in projecting distributed cognitive systems.

AB - Evaluation of the appropriateness of information technical systems for complex professional usage in safety-critical contexts poses significant methodical and practical challenges. In this study, the usability of a Safety Information and Alarm Panel (SIAP) in a nuclear power plant control room was tested. An integrated validation concept was used that included a new approach to measuring system and operator performance in complex work environments. The tested system was designed to aid the operators in severe disturbance and emergency situations. It had already been implemented at a nuclear power plant. The study was conducted in a full-scope training simulator. The results verified that an acceptable level of performance could be achieved when using the SIAP. When the operators’ practices were analysed by a habit-centred analysis, it was discovered that the effects of the SIAP differed between crews and between test scenarios. Thus, the SIAP tended to promote coherence of practices but reduce situatively attentive action. In diffuse task contexts the tool failed to support the shift supervisor's control of the overall process situation, his awareness of the crew's work load and his ability to update the crew's awareness of the process. The operators reported that the system supported their process control activity and reduced stress in the situation, but the shift supervisors and operators also noticed some possible negative effects of the tool. These subjective evaluations corresponded to the effects observed in practice. The results revealed the complexity of the implementation of new tools into professional practice. It was proposed that a validation project should focus on the trajectory of development of the entire distributed cognitive system instead of comprehending validation studies as tests of the effects of information systems on a pre-defined process output. Formative evaluation criteria are needed in projecting distributed cognitive systems.

KW - validation method

KW - information presentation

KW - control room

KW - operators

KW - practice

KW - nuclear power plants

KW - SIAP

KW - nuclear reactor safety

KW - safety

KW - alarm systems

U2 - 10.1016/j.ijhcs.2005.03.004

DO - 10.1016/j.ijhcs.2005.03.004

M3 - Article

VL - 63

SP - 328

EP - 361

JO - International Journal of Human-Computer Studies

JF - International Journal of Human-Computer Studies

SN - 1071-5819

IS - 3

ER -