Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II

Yrjö Virtanen, Sirpa Torkkeli, Bob Wilson

Research output: Book/ReportReport

Abstract

Because of the complexity and trade-offs between different points of the life cycles of the analysed systems, a method which measures the environmental damage caused by each intervention is needed in order to make a choice between the products. However, there is no commonly agreed methodology for this particular purpose. In most of the methods the valuation is implicitly or explicitly based on economic criteria. For various reasons, however, economically obtained criteria do not necessarily reflect ecological arguments correctly. Thus, there is a need for new, ecologically based valuation methods. One such approach is the expert judgement method, based on the Delphi technique, which rejects the economic basis in favour of the judgements of a group of environmental experts. However, it is not self evident that the expert judgement based environmental rating of interventions will be essentially more correct and certain than other methods. In this study the method was evaluated at different points of the procedure in order to obtain a picture of the quality of the indexes produced. The evaluation was based on an actual Delphi study made in 1995-1996 in Finland, Sweden and Norway. The main questions addressed were the significance of the results and the operational quality of the Delphi procedure. The results obtained by applying the expert method indexes were also compared with the results obtained with other valuation methods for the background life cycle inventory of the case study. Additional material included feedback data from panellists of the case study, collected with a questionnaire. The questionnaire data was analysed to identify major dimensions in the criteria for evaluating interventions and correlation of the final indexes of the Delphi I study with these dimensions. The rest of the questionnaire material was used to document panellists' opinions and experiences of the Delphi process, familiarity with the environmental impacts of various interventions, and classification in typologies of cultural theory. The quality of results and methodological aspects, such as effects of task instructions, selection of the index basis, and effects of the final standardisation were analysed statistically. Accordingly, the effects of various postulates made on the conformity of the environmental harm conceptions of the experts, and the influence of the moderators' decisions were assessed on the basis of standard statistical indicators. The state of consensus and its development in the Delphi process were studied with the aid of K-entropy analysis. The study showed that transparency and certainty, which are essential qualities for an acceptable and trusted valuation method, are only partially accomplished by the expert judgement method in the format in which it was developed in the analysed case. As for the technical procedure, the method is well documented and transparency is good. Argumentation of the judgements, however, should be increased. The quality of the valuation indexes is explicitly available, but their certainty is very low for most interventions. The opinions of the experts vary greatly. How much this depends on different values and how much on differences in knowledge etc. is impossible to assess. Also, how much the technique used and the statistical processing of the experts' answers may have influenced the eventual scores of different interventions is difficult to assess. The application of expert judgement to LCA valuation is a new idea, and the method is still very much under development and far from maturity. Nevertheless, utilisation of expert knowledge can be a significant addition to model approaches to ecological impact assessment, which, because of the chaotic behaviour of ecosystems, are limited and uncertain in predicting the ecological consequences of interventions to the environment. This should be taken into account when considering the results of the evaluation of the case study, which was the third of its kind in Europe.
Original languageEnglish
Place of PublicationEspoo
PublisherVTT Technical Research Centre of Finland
Number of pages142
ISBN (Electronic)951-38-5462-0
ISBN (Print)951-38-5461-2
Publication statusPublished - 1999
MoE publication typeD4 Published development or research report or study

Publication series

SeriesVTT Tiedotteita - Meddelanden - Research Notes
Number1972
ISSN1235-0605

Fingerprint

valuation
transparency
method
evaluation
life cycle
index method
familiarity
ecological impact
standardization
economics
typology
entropy
environmental impact
methodology
index
ecosystem

Keywords

  • LCA
  • Life Cycle Assessment
  • Delphi method
  • DELPHI II
  • valuation
  • environmental impacts
  • statistical analysis

Cite this

Virtanen, Y., Torkkeli, S., & Wilson, B. (1999). Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II. Espoo: VTT Technical Research Centre of Finland. VTT Tiedotteita - Meddelanden - Research Notes, No. 1972
Virtanen, Yrjö ; Torkkeli, Sirpa ; Wilson, Bob. / Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II. Espoo : VTT Technical Research Centre of Finland, 1999. 142 p. (VTT Tiedotteita - Meddelanden - Research Notes; No. 1972).
@book{0d5db3db9f6941f98af446a25cf2e105,
title = "Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II",
abstract = "Because of the complexity and trade-offs between different points of the life cycles of the analysed systems, a method which measures the environmental damage caused by each intervention is needed in order to make a choice between the products. However, there is no commonly agreed methodology for this particular purpose. In most of the methods the valuation is implicitly or explicitly based on economic criteria. For various reasons, however, economically obtained criteria do not necessarily reflect ecological arguments correctly. Thus, there is a need for new, ecologically based valuation methods. One such approach is the expert judgement method, based on the Delphi technique, which rejects the economic basis in favour of the judgements of a group of environmental experts. However, it is not self evident that the expert judgement based environmental rating of interventions will be essentially more correct and certain than other methods. In this study the method was evaluated at different points of the procedure in order to obtain a picture of the quality of the indexes produced. The evaluation was based on an actual Delphi study made in 1995-1996 in Finland, Sweden and Norway. The main questions addressed were the significance of the results and the operational quality of the Delphi procedure. The results obtained by applying the expert method indexes were also compared with the results obtained with other valuation methods for the background life cycle inventory of the case study. Additional material included feedback data from panellists of the case study, collected with a questionnaire. The questionnaire data was analysed to identify major dimensions in the criteria for evaluating interventions and correlation of the final indexes of the Delphi I study with these dimensions. The rest of the questionnaire material was used to document panellists' opinions and experiences of the Delphi process, familiarity with the environmental impacts of various interventions, and classification in typologies of cultural theory. The quality of results and methodological aspects, such as effects of task instructions, selection of the index basis, and effects of the final standardisation were analysed statistically. Accordingly, the effects of various postulates made on the conformity of the environmental harm conceptions of the experts, and the influence of the moderators' decisions were assessed on the basis of standard statistical indicators. The state of consensus and its development in the Delphi process were studied with the aid of K-entropy analysis. The study showed that transparency and certainty, which are essential qualities for an acceptable and trusted valuation method, are only partially accomplished by the expert judgement method in the format in which it was developed in the analysed case. As for the technical procedure, the method is well documented and transparency is good. Argumentation of the judgements, however, should be increased. The quality of the valuation indexes is explicitly available, but their certainty is very low for most interventions. The opinions of the experts vary greatly. How much this depends on different values and how much on differences in knowledge etc. is impossible to assess. Also, how much the technique used and the statistical processing of the experts' answers may have influenced the eventual scores of different interventions is difficult to assess. The application of expert judgement to LCA valuation is a new idea, and the method is still very much under development and far from maturity. Nevertheless, utilisation of expert knowledge can be a significant addition to model approaches to ecological impact assessment, which, because of the chaotic behaviour of ecosystems, are limited and uncertain in predicting the ecological consequences of interventions to the environment. This should be taken into account when considering the results of the evaluation of the case study, which was the third of its kind in Europe.",
keywords = "LCA, Life Cycle Assessment, Delphi method, DELPHI II, valuation, environmental impacts, statistical analysis",
author = "Yrj{\"o} Virtanen and Sirpa Torkkeli and Bob Wilson",
note = "Project code: K7SU00054",
year = "1999",
language = "English",
isbn = "951-38-5461-2",
series = "VTT Tiedotteita - Meddelanden - Research Notes",
publisher = "VTT Technical Research Centre of Finland",
number = "1972",
address = "Finland",

}

Virtanen, Y, Torkkeli, S & Wilson, B 1999, Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II. VTT Tiedotteita - Meddelanden - Research Notes, no. 1972, VTT Technical Research Centre of Finland, Espoo.

Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II. / Virtanen, Yrjö; Torkkeli, Sirpa; Wilson, Bob.

Espoo : VTT Technical Research Centre of Finland, 1999. 142 p. (VTT Tiedotteita - Meddelanden - Research Notes; No. 1972).

Research output: Book/ReportReport

TY - BOOK

T1 - Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II

AU - Virtanen, Yrjö

AU - Torkkeli, Sirpa

AU - Wilson, Bob

N1 - Project code: K7SU00054

PY - 1999

Y1 - 1999

N2 - Because of the complexity and trade-offs between different points of the life cycles of the analysed systems, a method which measures the environmental damage caused by each intervention is needed in order to make a choice between the products. However, there is no commonly agreed methodology for this particular purpose. In most of the methods the valuation is implicitly or explicitly based on economic criteria. For various reasons, however, economically obtained criteria do not necessarily reflect ecological arguments correctly. Thus, there is a need for new, ecologically based valuation methods. One such approach is the expert judgement method, based on the Delphi technique, which rejects the economic basis in favour of the judgements of a group of environmental experts. However, it is not self evident that the expert judgement based environmental rating of interventions will be essentially more correct and certain than other methods. In this study the method was evaluated at different points of the procedure in order to obtain a picture of the quality of the indexes produced. The evaluation was based on an actual Delphi study made in 1995-1996 in Finland, Sweden and Norway. The main questions addressed were the significance of the results and the operational quality of the Delphi procedure. The results obtained by applying the expert method indexes were also compared with the results obtained with other valuation methods for the background life cycle inventory of the case study. Additional material included feedback data from panellists of the case study, collected with a questionnaire. The questionnaire data was analysed to identify major dimensions in the criteria for evaluating interventions and correlation of the final indexes of the Delphi I study with these dimensions. The rest of the questionnaire material was used to document panellists' opinions and experiences of the Delphi process, familiarity with the environmental impacts of various interventions, and classification in typologies of cultural theory. The quality of results and methodological aspects, such as effects of task instructions, selection of the index basis, and effects of the final standardisation were analysed statistically. Accordingly, the effects of various postulates made on the conformity of the environmental harm conceptions of the experts, and the influence of the moderators' decisions were assessed on the basis of standard statistical indicators. The state of consensus and its development in the Delphi process were studied with the aid of K-entropy analysis. The study showed that transparency and certainty, which are essential qualities for an acceptable and trusted valuation method, are only partially accomplished by the expert judgement method in the format in which it was developed in the analysed case. As for the technical procedure, the method is well documented and transparency is good. Argumentation of the judgements, however, should be increased. The quality of the valuation indexes is explicitly available, but their certainty is very low for most interventions. The opinions of the experts vary greatly. How much this depends on different values and how much on differences in knowledge etc. is impossible to assess. Also, how much the technique used and the statistical processing of the experts' answers may have influenced the eventual scores of different interventions is difficult to assess. The application of expert judgement to LCA valuation is a new idea, and the method is still very much under development and far from maturity. Nevertheless, utilisation of expert knowledge can be a significant addition to model approaches to ecological impact assessment, which, because of the chaotic behaviour of ecosystems, are limited and uncertain in predicting the ecological consequences of interventions to the environment. This should be taken into account when considering the results of the evaluation of the case study, which was the third of its kind in Europe.

AB - Because of the complexity and trade-offs between different points of the life cycles of the analysed systems, a method which measures the environmental damage caused by each intervention is needed in order to make a choice between the products. However, there is no commonly agreed methodology for this particular purpose. In most of the methods the valuation is implicitly or explicitly based on economic criteria. For various reasons, however, economically obtained criteria do not necessarily reflect ecological arguments correctly. Thus, there is a need for new, ecologically based valuation methods. One such approach is the expert judgement method, based on the Delphi technique, which rejects the economic basis in favour of the judgements of a group of environmental experts. However, it is not self evident that the expert judgement based environmental rating of interventions will be essentially more correct and certain than other methods. In this study the method was evaluated at different points of the procedure in order to obtain a picture of the quality of the indexes produced. The evaluation was based on an actual Delphi study made in 1995-1996 in Finland, Sweden and Norway. The main questions addressed were the significance of the results and the operational quality of the Delphi procedure. The results obtained by applying the expert method indexes were also compared with the results obtained with other valuation methods for the background life cycle inventory of the case study. Additional material included feedback data from panellists of the case study, collected with a questionnaire. The questionnaire data was analysed to identify major dimensions in the criteria for evaluating interventions and correlation of the final indexes of the Delphi I study with these dimensions. The rest of the questionnaire material was used to document panellists' opinions and experiences of the Delphi process, familiarity with the environmental impacts of various interventions, and classification in typologies of cultural theory. The quality of results and methodological aspects, such as effects of task instructions, selection of the index basis, and effects of the final standardisation were analysed statistically. Accordingly, the effects of various postulates made on the conformity of the environmental harm conceptions of the experts, and the influence of the moderators' decisions were assessed on the basis of standard statistical indicators. The state of consensus and its development in the Delphi process were studied with the aid of K-entropy analysis. The study showed that transparency and certainty, which are essential qualities for an acceptable and trusted valuation method, are only partially accomplished by the expert judgement method in the format in which it was developed in the analysed case. As for the technical procedure, the method is well documented and transparency is good. Argumentation of the judgements, however, should be increased. The quality of the valuation indexes is explicitly available, but their certainty is very low for most interventions. The opinions of the experts vary greatly. How much this depends on different values and how much on differences in knowledge etc. is impossible to assess. Also, how much the technique used and the statistical processing of the experts' answers may have influenced the eventual scores of different interventions is difficult to assess. The application of expert judgement to LCA valuation is a new idea, and the method is still very much under development and far from maturity. Nevertheless, utilisation of expert knowledge can be a significant addition to model approaches to ecological impact assessment, which, because of the chaotic behaviour of ecosystems, are limited and uncertain in predicting the ecological consequences of interventions to the environment. This should be taken into account when considering the results of the evaluation of the case study, which was the third of its kind in Europe.

KW - LCA

KW - Life Cycle Assessment

KW - Delphi method

KW - DELPHI II

KW - valuation

KW - environmental impacts

KW - statistical analysis

M3 - Report

SN - 951-38-5461-2

T3 - VTT Tiedotteita - Meddelanden - Research Notes

BT - Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II

PB - VTT Technical Research Centre of Finland

CY - Espoo

ER -

Virtanen Y, Torkkeli S, Wilson B. Evaluation of a Delphi technique based expert judgement method for LCA valuation DELPHI II. Espoo: VTT Technical Research Centre of Finland, 1999. 142 p. (VTT Tiedotteita - Meddelanden - Research Notes; No. 1972).