Evaluation report for optimal data quality in selected European service cases - QUANTIS Deliverable D6

This deliverable documents the results obtained in work package WP600 of QUANTIS (Quality Assessment and Assurance Methodology for Traffic Data and Information Services). The main objectives of WP600 are to evaluate service quality in the selected European service cases using the methodology developed during the QUANTIS project and to carry out socio-economic cost-benefit evaluations to find out the optimum service quality. Work package WP600 is essentially based on earlier work done within the QUANTIS project. The quality levels for the European ITS services have been described in work package QUANTIS WP400, the method used to evaluate service quality is the main product of WP500 and work packages WP200 and WP300 have documented the European service cases under analysis, provided the definitions of the most important concepts and explored the relations between service quality and costs and service quality and impacts. The evaluation of service quality was carried out for three service cases using the QUANTIS methodology. The total service quality is expressed as a percentage and a quality level which may have values between one and four of which level one represents only limited and level four the best level of service quality. The values of total service quality achieved by the services were 92,00%, 79,95% and 82,71%, which correspond to levels three for the first service case and level two for the remaining two service cases. The QUANTIS methodology provided meaningful results and a quantitative estimate of service quality for all three analysed European service cases. In some cases, minor adjustments to the quality levels included in the toolkit were needed. There were also some cases where it was challenging to find a meaningful interpretation for the quality attribute in question and differences between services classified under the definition of the same European ITS service. The evaluator using the QUANTIS toolkit should document the definitions of the quality attributes he has used and the way he has obtained the quantitative values for quality attributes from service description, expert interviews or field tests. In total, nine evaluations were carried out using socio-economic cost-benefit analysis to find the optimum service quality. The quality levels defined in QUANTIS WP400 were used directly in six evaluations and the highest quality level provided best benefit-cost ratio in three of them. In the remaining three evaluations one evaluation indicated the second highest level, level three, as the optimum quality level and in two evaluations the quality level with highest benefit-cost ratio was found to be 2. Because of the limited availability of empirical data and complexity of the system under analysis, two custom quality levels (error probabilities of 14% and 11% and service up-times of 98% and 99%) were analysed in the three evaluations of the Finnish service case. The higher quality level of error probability was found to have significantly better benefit-cost ratio in the Finnish service case. In general, higher levels of service quality seemed to provide better benefit-cost ratios. The limited availability of earlier evaluation studies affected the scope of socio-economic cost-benefit evaluations carried out in QUANTIS and the accuracy of evaluation results. Because there were no earlier evaluation studies available for some services analysed in QUANTIS, only the most important impacts could be analysed in these cases.