Software testing with code-based test generators

data and lessons learned from a case study with an industrial software component

Pietro Braione (Corresponding Author), Giovanni Denaro, Andrea Mattavelli, Mattia Vivanti, Ali Muhammad

Research output: Contribution to journalArticleScientificpeer-review

14 Citations (Scopus)

Abstract

Automatically generating effective test suites promises a significant impact on testing practice by promoting extensively tested software within reasonable effort and cost bounds. Code-based test generators rely on the source code of the software under test to identify test objectives and to steer the test case generation process accordingly. Currently, the most mature proposals on this topic come from the research on random testing, dynamic symbolic execution, and search-based testing. This paper studies the effectiveness of a set of state-of-the-research test generators on a family of industrial programs with nontrivial domain-specific peculiarities. These programs are part of a software component of a real-time and safety-critical control system and integrate in a control task specified in LabVIEW, a graphical language for designing embedded systems. The result of this study enhances the available body of knowledge on the strengths and weaknesses of test generators. The empirical data indicate that the test generators can truly expose subtle (previously unknown) bugs in the subject software and that there can be merit in using different types of test generation approaches in a complementary, even synergic fashion. Furthermore, our experiment pinpoints the support for floating point arithmetics and nonlinear computations as a major milestone in the path to exploiting the full potential of the prototypes based on symbolic execution in industry.
Original languageEnglish
Pages (from-to)311-333
Number of pages23
JournalSoftware Quality Journal
Volume22
Issue number2
DOIs
Publication statusPublished - 2014
MoE publication typeA1 Journal article-refereed

Fingerprint

Software testing
Testing
Digital arithmetic
Embedded systems
Control systems
Costs
Industry
Experiments

Keywords

  • structural testing
  • automatic test generation
  • experimental study

Cite this

Braione, Pietro ; Denaro, Giovanni ; Mattavelli, Andrea ; Vivanti, Mattia ; Muhammad, Ali. / Software testing with code-based test generators : data and lessons learned from a case study with an industrial software component. In: Software Quality Journal. 2014 ; Vol. 22, No. 2. pp. 311-333.
@article{e54880ff586e489095047aac85820633,
title = "Software testing with code-based test generators: data and lessons learned from a case study with an industrial software component",
abstract = "Automatically generating effective test suites promises a significant impact on testing practice by promoting extensively tested software within reasonable effort and cost bounds. Code-based test generators rely on the source code of the software under test to identify test objectives and to steer the test case generation process accordingly. Currently, the most mature proposals on this topic come from the research on random testing, dynamic symbolic execution, and search-based testing. This paper studies the effectiveness of a set of state-of-the-research test generators on a family of industrial programs with nontrivial domain-specific peculiarities. These programs are part of a software component of a real-time and safety-critical control system and integrate in a control task specified in LabVIEW, a graphical language for designing embedded systems. The result of this study enhances the available body of knowledge on the strengths and weaknesses of test generators. The empirical data indicate that the test generators can truly expose subtle (previously unknown) bugs in the subject software and that there can be merit in using different types of test generation approaches in a complementary, even synergic fashion. Furthermore, our experiment pinpoints the support for floating point arithmetics and nonlinear computations as a major milestone in the path to exploiting the full potential of the prototypes based on symbolic execution in industry.",
keywords = "structural testing, automatic test generation, experimental study",
author = "Pietro Braione and Giovanni Denaro and Andrea Mattavelli and Mattia Vivanti and Ali Muhammad",
year = "2014",
doi = "10.1007/s11219-013-9207-1",
language = "English",
volume = "22",
pages = "311--333",
journal = "Software Quality Journal",
issn = "0963-9314",
publisher = "Springer",
number = "2",

}

Software testing with code-based test generators : data and lessons learned from a case study with an industrial software component. / Braione, Pietro (Corresponding Author); Denaro, Giovanni; Mattavelli, Andrea; Vivanti, Mattia; Muhammad, Ali.

In: Software Quality Journal, Vol. 22, No. 2, 2014, p. 311-333.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Software testing with code-based test generators

T2 - data and lessons learned from a case study with an industrial software component

AU - Braione, Pietro

AU - Denaro, Giovanni

AU - Mattavelli, Andrea

AU - Vivanti, Mattia

AU - Muhammad, Ali

PY - 2014

Y1 - 2014

N2 - Automatically generating effective test suites promises a significant impact on testing practice by promoting extensively tested software within reasonable effort and cost bounds. Code-based test generators rely on the source code of the software under test to identify test objectives and to steer the test case generation process accordingly. Currently, the most mature proposals on this topic come from the research on random testing, dynamic symbolic execution, and search-based testing. This paper studies the effectiveness of a set of state-of-the-research test generators on a family of industrial programs with nontrivial domain-specific peculiarities. These programs are part of a software component of a real-time and safety-critical control system and integrate in a control task specified in LabVIEW, a graphical language for designing embedded systems. The result of this study enhances the available body of knowledge on the strengths and weaknesses of test generators. The empirical data indicate that the test generators can truly expose subtle (previously unknown) bugs in the subject software and that there can be merit in using different types of test generation approaches in a complementary, even synergic fashion. Furthermore, our experiment pinpoints the support for floating point arithmetics and nonlinear computations as a major milestone in the path to exploiting the full potential of the prototypes based on symbolic execution in industry.

AB - Automatically generating effective test suites promises a significant impact on testing practice by promoting extensively tested software within reasonable effort and cost bounds. Code-based test generators rely on the source code of the software under test to identify test objectives and to steer the test case generation process accordingly. Currently, the most mature proposals on this topic come from the research on random testing, dynamic symbolic execution, and search-based testing. This paper studies the effectiveness of a set of state-of-the-research test generators on a family of industrial programs with nontrivial domain-specific peculiarities. These programs are part of a software component of a real-time and safety-critical control system and integrate in a control task specified in LabVIEW, a graphical language for designing embedded systems. The result of this study enhances the available body of knowledge on the strengths and weaknesses of test generators. The empirical data indicate that the test generators can truly expose subtle (previously unknown) bugs in the subject software and that there can be merit in using different types of test generation approaches in a complementary, even synergic fashion. Furthermore, our experiment pinpoints the support for floating point arithmetics and nonlinear computations as a major milestone in the path to exploiting the full potential of the prototypes based on symbolic execution in industry.

KW - structural testing

KW - automatic test generation

KW - experimental study

U2 - 10.1007/s11219-013-9207-1

DO - 10.1007/s11219-013-9207-1

M3 - Article

VL - 22

SP - 311

EP - 333

JO - Software Quality Journal

JF - Software Quality Journal

SN - 0963-9314

IS - 2

ER -