This month Health In Harmony Research Director, Bethany Kois, interviewed Dr. Gill Westhorp, an Australian researcher who is designing the realist approach survey to evaluate our model at Project ASRI.
Bethany Kois: Tell me about realist methodology?
Gill Westhorp: That’s a big question! It’s called ‘realist’ because it’s based in a realist philosophy of science. That has implications for what we think about the nature of the world, the nature of knowledge, and how we can go about researching it. The most important thing from a program point of view is about how programs cause change. Realists don’t believe that programs cause change directly. Programs provide resources and opportunities, and sometimes constraints (“You can’t do this because…”). It’s the decisions that people make in response to those resources and opportunities that determine what happens next, and the outcomes of the program. People in different circumstances and with different beliefs make different decisions, so you get different outcomes for different people. So realist evaluation tries to understand the ‘reasoning’ that leads to different decisions and the factors about the context that influence those decisions. Realists don’t ask “Does it work?” We ask “For whom does it work, in what contexts, in what respects, to what extent, and how?”
BK: In your opinion, as a researcher, what makes a good evaluation project?
GW: I come from a service delivery and policy background, so for me the big thing is whether the results will be used. From a researcher perspective, that means my job is to make the results useful. That means I have to understand what the client wants to do with the results and that shapes the nature of the questions that the evaluation is built around. The questions need to be clear, and then the evaluation needs to be designed to provide strong evidence in relation to those questions. You can never answer everything you want to, so you aim do a good job on what you tackle and to be honest about the limitations of what you’ve been able to do.
What makes it interesting for me is if it’s challenging to design the evaluation. I like the problem solving involved in working out how to do it as much as I like finding out how and why programs work the way they do.
BK: What would you like Health In Harmony (HIH) supporters to understand about the realist evaluation you just designed for us?
GW: That it’s the beginning of a new stage in HIH learning. It’s not the beginning of HIH doing research or evaluation – there’s an admirable history of that. Taking a realist approach is new though. This project is particularly designed to help us understand the “how and why” the HIH model works and doesn’t work for different people in different circumstances. The idea is that we’ll use some of what we learn to adapt the design of next year’s evaluation work, and it’s that next stage that will tell us more about the outcomes (for whom, in what respects and so on).
There are interesting possibilities about working with another organisation that does similar work, but based around marine environments in Madagascar, to build research and evaluation that helps understand this kind of work in very different cultural and environmental perspectives. We’re looking for funding for that now. If we’re successful, that will add new layers of learning and new layers of value.
BK: What excites you about this evaluation?
GW: What excites me most was the questions that HIH wants to answer in the longer term. Asking genuine questions about – well there seem to be changes happening where we’re working, but how much of that is a result of what we’re doing and how much of it is down to other factors? And – it looks as though we are contributing to some pretty useful changes, but what is it about what we do and the way we do it that makes it work? Those questions reflect both a rigorous approach to evaluation research and a genuine commitment to doing the work of the organisation as well as it can be done. That’s a great combination.
The thing I’m enjoying the most is the people I’m working with in the evaluation. Great team, genuine warmth, and a real commitment to working together. You don’t always find that when you’re working with people you’ve never met face to face!
BK: What benefits do you see coming out of this work?
GW: I hope there’ll be benefits on a couple of different levels. One is about what we’ll find out about how and why and for whom the program works – and why not when it doesn’t work, and for whom not. Those “negative” findings are often the most useful for really understanding the program – you need the comparison of when it does and when it doesn’t work to do the analysis. They’re also the most useful for program improvement. So in the short term, learning and in the medium term, use of that learning to improve effectiveness, and to know what to do to extend the program to other places.
The other benefit - I hope - is that some people in the organisation will have learnt a bit about this way of going about evaluation and will be able to use it in future. I really think that using a realist approach does change what one learns from an evaluation, and that changes how useful it is.