Evaluation report for optimal data quality in selected European service cases

Risto Öörni, Satu Innamaa, Risto Kulmala, Astrid Kellermann, Robert Ebner, Doug Newton

Research output: Book/ReportReport

Abstract

This deliverable documents the results obtained in work package WP600 of QUANTIS (Quality Assessment and Assurance Methodology for Traffic Data and Information Services). The main objectives of WP600 are to evaluate service quality in the selected European service cases using the methodology developed during the QUANTIS project and to carry out socio-economic cost-benefit evaluations to find out the optimum service quality. Work package WP600 is essentially based on earlier work done within the QUANTIS project. The quality levels for the European ITS services have been described in work package QUANTIS WP400, the method used to evaluate service quality is the main product of WP500 and work packages WP200 and WP300 have documented the European service cases under analysis, provided the definitions of the most important concepts and explored the relations between service quality and costs and service quality and impacts. The evaluation of service quality was carried out for three service cases using the QUANTIS methodology. The total service quality is expressed as a percentage and a quality level which may have values between one and four of which level one represents only limited and level four the best level of service quality. The values of total service quality achieved by the services were 92,00%, 79,95% and 82,71%, which correspond to levels three for the first service case and level two for the remaining two service cases. The QUANTIS methodology provided meaningful results and a quantitative estimate of service quality for all three analysed European service cases. In some cases, minor adjustments to the quality levels included in the toolkit were needed. There were also some cases where it was challenging to find a meaningful interpretation for the quality attribute in question and differences between services classified under the definition of the same European ITS service. The evaluator using the QUANTIS toolkit should document the definitions of the quality attributes he has used and the way he has obtained the quantitative values for quality attributes from service description, expert interviews or field tests. In total, nine evaluations were carried out using socio-economic cost-benefit analysis to find the optimum service quality. The quality levels defined in QUANTIS WP400 were used directly in six evaluations and the highest quality level provided best benefit-cost ratio in three of them. In the remaining three evaluations one evaluation indicated the second highest level, level three, as the optimum quality level and in two evaluations the quality level with highest benefit-cost ratio was found to be 2. Because of the limited availability of empirical data and complexity of the system under analysis, two custom quality levels (error probabilities of 14% and 11% and service up-times of 98% and 99%) were analysed in the three evaluations of the Finnish service case. The higher quality level of error probability was found to have significantly better benefit-cost ratio in the Finnish service case. In general, higher levels of service quality seemed to provide better benefit-cost ratios. The limited availability of earlier evaluation studies affected the scope of socio-economic cost-benefit evaluations carried out in QUANTIS and the accuracy of evaluation results. Because there were no earlier evaluation studies available for some services analysed in QUANTIS, only the most important impacts could be analysed in these cases.
Original languageEnglish
Number of pages88
Publication statusPublished - 2010
MoE publication typeD4 Published development or research report or study

Publication series

SeriesQUANTIS Project reports
NumberDeliverable D6

Fingerprint

Data quality
Evaluation
Service quality
Benefit-cost ratio
Methodology
Quality attributes
Socio-economics
Economic cost
Toolkit
Quality of service
Level of service
Cost-benefit
Cost-benefit analysis
Information services
Empirical data
Quality assessment
Costs
Quality assurance
Evaluator

Keywords

  • ITS
  • data quality
  • service quality

Cite this

Öörni, R., Innamaa, S., Kulmala, R., Kellermann, A., Ebner, R., & Newton, D. (2010). Evaluation report for optimal data quality in selected European service cases. QUANTIS Project reports, No. Deliverable D6
Öörni, Risto ; Innamaa, Satu ; Kulmala, Risto ; Kellermann, Astrid ; Ebner, Robert ; Newton, Doug. / Evaluation report for optimal data quality in selected European service cases. 2010. 88 p. (QUANTIS Project reports; No. Deliverable D6).
@book{7a7dd579a8f1425b85637b12171ecdda,
title = "Evaluation report for optimal data quality in selected European service cases",
abstract = "This deliverable documents the results obtained in work package WP600 of QUANTIS (Quality Assessment and Assurance Methodology for Traffic Data and Information Services). The main objectives of WP600 are to evaluate service quality in the selected European service cases using the methodology developed during the QUANTIS project and to carry out socio-economic cost-benefit evaluations to find out the optimum service quality. Work package WP600 is essentially based on earlier work done within the QUANTIS project. The quality levels for the European ITS services have been described in work package QUANTIS WP400, the method used to evaluate service quality is the main product of WP500 and work packages WP200 and WP300 have documented the European service cases under analysis, provided the definitions of the most important concepts and explored the relations between service quality and costs and service quality and impacts. The evaluation of service quality was carried out for three service cases using the QUANTIS methodology. The total service quality is expressed as a percentage and a quality level which may have values between one and four of which level one represents only limited and level four the best level of service quality. The values of total service quality achieved by the services were 92,00{\%}, 79,95{\%} and 82,71{\%}, which correspond to levels three for the first service case and level two for the remaining two service cases. The QUANTIS methodology provided meaningful results and a quantitative estimate of service quality for all three analysed European service cases. In some cases, minor adjustments to the quality levels included in the toolkit were needed. There were also some cases where it was challenging to find a meaningful interpretation for the quality attribute in question and differences between services classified under the definition of the same European ITS service. The evaluator using the QUANTIS toolkit should document the definitions of the quality attributes he has used and the way he has obtained the quantitative values for quality attributes from service description, expert interviews or field tests. In total, nine evaluations were carried out using socio-economic cost-benefit analysis to find the optimum service quality. The quality levels defined in QUANTIS WP400 were used directly in six evaluations and the highest quality level provided best benefit-cost ratio in three of them. In the remaining three evaluations one evaluation indicated the second highest level, level three, as the optimum quality level and in two evaluations the quality level with highest benefit-cost ratio was found to be 2. Because of the limited availability of empirical data and complexity of the system under analysis, two custom quality levels (error probabilities of 14{\%} and 11{\%} and service up-times of 98{\%} and 99{\%}) were analysed in the three evaluations of the Finnish service case. The higher quality level of error probability was found to have significantly better benefit-cost ratio in the Finnish service case. In general, higher levels of service quality seemed to provide better benefit-cost ratios. The limited availability of earlier evaluation studies affected the scope of socio-economic cost-benefit evaluations carried out in QUANTIS and the accuracy of evaluation results. Because there were no earlier evaluation studies available for some services analysed in QUANTIS, only the most important impacts could be analysed in these cases.",
keywords = "ITS, data quality, service quality",
author = "Risto {\"O}{\"o}rni and Satu Innamaa and Risto Kulmala and Astrid Kellermann and Robert Ebner and Doug Newton",
note = "Project code: 23048",
year = "2010",
language = "English",
series = "QUANTIS Project reports",
number = "Deliverable D6",

}

Öörni, R, Innamaa, S, Kulmala, R, Kellermann, A, Ebner, R & Newton, D 2010, Evaluation report for optimal data quality in selected European service cases. QUANTIS Project reports, no. Deliverable D6.

Evaluation report for optimal data quality in selected European service cases. / Öörni, Risto; Innamaa, Satu; Kulmala, Risto; Kellermann, Astrid; Ebner, Robert; Newton, Doug.

2010. 88 p. (QUANTIS Project reports; No. Deliverable D6).

Research output: Book/ReportReport

TY - BOOK

T1 - Evaluation report for optimal data quality in selected European service cases

AU - Öörni, Risto

AU - Innamaa, Satu

AU - Kulmala, Risto

AU - Kellermann, Astrid

AU - Ebner, Robert

AU - Newton, Doug

N1 - Project code: 23048

PY - 2010

Y1 - 2010

N2 - This deliverable documents the results obtained in work package WP600 of QUANTIS (Quality Assessment and Assurance Methodology for Traffic Data and Information Services). The main objectives of WP600 are to evaluate service quality in the selected European service cases using the methodology developed during the QUANTIS project and to carry out socio-economic cost-benefit evaluations to find out the optimum service quality. Work package WP600 is essentially based on earlier work done within the QUANTIS project. The quality levels for the European ITS services have been described in work package QUANTIS WP400, the method used to evaluate service quality is the main product of WP500 and work packages WP200 and WP300 have documented the European service cases under analysis, provided the definitions of the most important concepts and explored the relations between service quality and costs and service quality and impacts. The evaluation of service quality was carried out for three service cases using the QUANTIS methodology. The total service quality is expressed as a percentage and a quality level which may have values between one and four of which level one represents only limited and level four the best level of service quality. The values of total service quality achieved by the services were 92,00%, 79,95% and 82,71%, which correspond to levels three for the first service case and level two for the remaining two service cases. The QUANTIS methodology provided meaningful results and a quantitative estimate of service quality for all three analysed European service cases. In some cases, minor adjustments to the quality levels included in the toolkit were needed. There were also some cases where it was challenging to find a meaningful interpretation for the quality attribute in question and differences between services classified under the definition of the same European ITS service. The evaluator using the QUANTIS toolkit should document the definitions of the quality attributes he has used and the way he has obtained the quantitative values for quality attributes from service description, expert interviews or field tests. In total, nine evaluations were carried out using socio-economic cost-benefit analysis to find the optimum service quality. The quality levels defined in QUANTIS WP400 were used directly in six evaluations and the highest quality level provided best benefit-cost ratio in three of them. In the remaining three evaluations one evaluation indicated the second highest level, level three, as the optimum quality level and in two evaluations the quality level with highest benefit-cost ratio was found to be 2. Because of the limited availability of empirical data and complexity of the system under analysis, two custom quality levels (error probabilities of 14% and 11% and service up-times of 98% and 99%) were analysed in the three evaluations of the Finnish service case. The higher quality level of error probability was found to have significantly better benefit-cost ratio in the Finnish service case. In general, higher levels of service quality seemed to provide better benefit-cost ratios. The limited availability of earlier evaluation studies affected the scope of socio-economic cost-benefit evaluations carried out in QUANTIS and the accuracy of evaluation results. Because there were no earlier evaluation studies available for some services analysed in QUANTIS, only the most important impacts could be analysed in these cases.

AB - This deliverable documents the results obtained in work package WP600 of QUANTIS (Quality Assessment and Assurance Methodology for Traffic Data and Information Services). The main objectives of WP600 are to evaluate service quality in the selected European service cases using the methodology developed during the QUANTIS project and to carry out socio-economic cost-benefit evaluations to find out the optimum service quality. Work package WP600 is essentially based on earlier work done within the QUANTIS project. The quality levels for the European ITS services have been described in work package QUANTIS WP400, the method used to evaluate service quality is the main product of WP500 and work packages WP200 and WP300 have documented the European service cases under analysis, provided the definitions of the most important concepts and explored the relations between service quality and costs and service quality and impacts. The evaluation of service quality was carried out for three service cases using the QUANTIS methodology. The total service quality is expressed as a percentage and a quality level which may have values between one and four of which level one represents only limited and level four the best level of service quality. The values of total service quality achieved by the services were 92,00%, 79,95% and 82,71%, which correspond to levels three for the first service case and level two for the remaining two service cases. The QUANTIS methodology provided meaningful results and a quantitative estimate of service quality for all three analysed European service cases. In some cases, minor adjustments to the quality levels included in the toolkit were needed. There were also some cases where it was challenging to find a meaningful interpretation for the quality attribute in question and differences between services classified under the definition of the same European ITS service. The evaluator using the QUANTIS toolkit should document the definitions of the quality attributes he has used and the way he has obtained the quantitative values for quality attributes from service description, expert interviews or field tests. In total, nine evaluations were carried out using socio-economic cost-benefit analysis to find the optimum service quality. The quality levels defined in QUANTIS WP400 were used directly in six evaluations and the highest quality level provided best benefit-cost ratio in three of them. In the remaining three evaluations one evaluation indicated the second highest level, level three, as the optimum quality level and in two evaluations the quality level with highest benefit-cost ratio was found to be 2. Because of the limited availability of empirical data and complexity of the system under analysis, two custom quality levels (error probabilities of 14% and 11% and service up-times of 98% and 99%) were analysed in the three evaluations of the Finnish service case. The higher quality level of error probability was found to have significantly better benefit-cost ratio in the Finnish service case. In general, higher levels of service quality seemed to provide better benefit-cost ratios. The limited availability of earlier evaluation studies affected the scope of socio-economic cost-benefit evaluations carried out in QUANTIS and the accuracy of evaluation results. Because there were no earlier evaluation studies available for some services analysed in QUANTIS, only the most important impacts could be analysed in these cases.

KW - ITS

KW - data quality

KW - service quality

M3 - Report

T3 - QUANTIS Project reports

BT - Evaluation report for optimal data quality in selected European service cases

ER -

Öörni R, Innamaa S, Kulmala R, Kellermann A, Ebner R, Newton D. Evaluation report for optimal data quality in selected European service cases. 2010. 88 p. (QUANTIS Project reports; No. Deliverable D6).