Enabling Software Process Improvement in Agile Software Development Teams and Organisations: Dissertation

Agile software development has challenged the traditional ways of delivering software as it provides a very different approach to software development. In recent decades, software process improvement (SPI) has been widely studied in the context of traditional software development, and its strengths and weaknesses have been recognised. As organisations increasingly adopt agile software development methodologies to be used alongside traditional methodologies, new challenges and opportunities for SPI are also emerging. One challenge is that traditional SPI methods often emphasise the continuous improvement of organisational software development processes, whereas the principles of agile software development focus on iterative adaptation and improvement of the activities of individual software development teams to increase effectiveness. The focus of this thesis is twofold. The first goal is to study how agile software development teams can conduct SPI, according to the values, principles and practices of agile software development, in tandem with the success factors of traditional SPI. The second goal is to study how the team-centred SPI of agile software development and the traditional view of organisational improvement can be integrated to co-exist in a mutually-beneficial manner in software development organisations. The main research methodology in this thesis is action research (AR). The empirical data is taken from six agile software development case projects. The results of this research have been published in a total of seven conference, and journal, papers. The principal result of the study of project level SPI is an Iterative Improvement Process which provides systematic, yet agile, SPI mechanisms for agile software development teams. This process iteratively evolved during the series of case projects. The empirical evidence of the project level research demonstrates the ability and willingness of agile software development teams to iteratively improve their daily working practices by making minor and simple, yet effective and visible, improvements during their projects. The research data further indicates the positive effect of iterative team reflection on the satisfaction of project teams, and confirms the need for systematic mechanisms to carry out SPI activities in agile project teams. Furthermore, the data shows that external support for the improvement activities proved to be highly significant for the success of SPI within agile project teams. The study of organizational SPI initially focused on integrating agile software development and continuous improvement of existing organisational practices. Arising from this stage, several changes to traditional SPI activities were suggested in order to establish a mutually-beneficial co-existence between organisational SPI stakeholders and agile software development teams. During the research a framework for deploying agile practices in organisations was developed. In this novel framework, the Iterative Improvement Process provides a mechanism for feedback between the agile software development teams and continuous organisational improvement activities. The research data further indicates that documented and validated knowledge arising from the Iterative Improvement Processes of agile software development teams can be beneficial in other contexts, such as in analysing and establishing future SPI initiatives in software development organisations.