A validation approach for product-focused process improvement

As the number of software engineering improvement methodologies and their adoption rate in industry increase, the validation of improvement methodologies becomes more and more important. Past validation studies show the effectiveness of improvement methodologies. However, they also reveal many technical difficulties for scientifically sound and detailed validation studies. This paper surveys the state of the art in improvement methodology validation and derives recommendations for systematic validation studies, which are substantiated by experiences from the European PROFES project. PROFES has developed a product-focused software process improvement methodology and started its empirical validation already early in the project. In brief, the main results of our validation work are: (1) Explicit product quality goals increase the effectiveness of process improvement and allow for causal analysis of observed improvements. (2) Validation should be based on explicit, a-priori set hypotheses involving multi-facetted validation criteria. (3) Improvement methodologies of different types, such as process assessment and goal-oriented measurement, are rather complementary than competing approaches.