Evaluation good practice: is ‘good enough’ better than ‘perfect’?

Panel: 8. Monitoring and evaluation: building confidence and enhancing practices

This is a peer-reviewed paper.

Authors:
Joanne Wade, Association for Decentralised Energy/ADE, United Kingdom
Nick Eyre, Environmental Change Institute, United Kingdom

Abstract

The ‘gold standard’ of perfect evaluation practice may in theory be defined as the robust implementation of a Randomised Control Trial (RCT). In reality, the implementation of such an approach has rarely been possible for energy efficiency programmes, and evaluators have delivered studies that are ‘good enough’, within constraints defined by programme design, evaluation budget and timeframe, and evaluation aims.

Drawing on a systematic review of the peer-reviewed literature on household energy efficiency evaluation undertaken for the UK Energy Research Centre, this paper debates priorities for future evaluation research, based on an analysis of possible gaps in knowledge and evaluation practice. It assesses the benefits and drawbacks of different evaluation methods (including RCT, quasi-experimental methods, and engineering estimates) in terms of cost, complexity and accuracy (in the context of impact evaluation of programmes or policies). It identifies the potential shortcomings of differing methods (e.g. reliance of engineering estimates on deemed savings and the availability of suitable sample sizes for experimental approaches). It sets outs some key gaps in our knowledge about the impacts of energy efficiency programmes, which pose new challenges for evaluation, including assessment of how impacts vary across end-users (rather than just average effects) and assessment of wider market transformation by large scale programmes.

It concludes that there is a role for a range of different evaluation approaches from rigorous RCT under tightly controlled conditions to information collection from major programmes, and that greater efforts are needed to share and debate existing information, via peer review and publication.