Aotearoa New Zealand National Evaluation Conference

Joseph Schumacher
Debi Mujumdar
Christina Howard
Geoff Stone
Salena Davie
Family, Child, Youth and Community Team
Centre for Social Research and Evaluation
Ministry of Social Development

The third annual Aotearoa New Zealand National Evaluation Conference, sponsored by the Auckland Evaluation Group and held in Taupo from 31 July to 2 August 2006, attracted over 50 participants from throughout New Zealand. Its theme was Evaluation and Social Change – What Are the Links? and there was particular interest in exploring bicultural and kaupapa Māori models of community and evaluation work.

The purpose of the conference was to bring professional evaluation practitioners together to share experiences, ideas and ways of working. Most of the participants at the first conference were evaluation contractors, but since then increasing numbers of evaluators from the state sector have been attending. The previous two conferences explored the desirability and feasibility of establishing a New Zealand-based professional evaluation association. This discussion – and the sustained efforts of a dedicated working group – led to the official launch of the Aotearoa New Zealand Evaluation Association, or anzea, at this year’s conference. Anzea will also be the official host of the next conference, planned for mid-2007.

The open forum at the end of the conference gave participants an opportunity to discuss the conference’s highlights and suggest themes and priorities for the next one. There was widespread endorsement of the longer sessions at this year’s conference because they allowed sufficient time for questions and discussion. Participants liked the interactive nature of the sessions and felt the conferences helped create a sense of community – a community of evaluation practice.

Over the conference’s three days, which included three keynote addresses and a dozen workshops, seminars and group discussions, participants explored the role of evaluation in social change. A range of issues and questions were raised, including:

• Do evaluators have a role as agents of social change?
• What are the values that determine the various roles evaluators undertake?
• How do we make connections and weave together the interests of different sectors?
• What is the role of the evaluator as contract provider within a purchaser/provider relationship?
• How can evaluators exert influence and is that appropriate?
• How do we build capacity for social change in evaluation?

Case Studies

Karen Sewell (CEO, Education Review Office and Secretary for Education), in the conference’s first keynote address, focused on the role of the Education Review Office (ERO) as an agent for social change. She argued that ERO, an independent government department, facilitates social change through evaluating the quality of pre-tertiary education in New Zealand; informing schools, parents, communities and policymakers in the government of its evaluation findings; highlighting the need to evaluate education in an indigenous context (taking the Māori worldview into account and working in collaboration with Te Kōhanga Reo and the Ministry of Education); and enabling three evaluation strands to be accommodated – compliance issues, areas of national interest and school priorities.

In the second keynote address, Gael Surgenor (Ministry of Social Development Project Manager for SKIP), Kaira Marsh-Goudi (SKIP Coordinator, Homebuilders Family Centre, Warkworth), Ripeka Kaipuke (Ngāti Pahauwera, Ngāti Kahungunu), and Bruce Maden (Te Aroha Noa Community Services, Palmerston North) talked of their experience with the SKIP Local Initiative Fund self-evaluation project. The discussion included a brief description of the Strategies for Kids, Information for Parents (SKIP) in terms of its background, objectives, funding and strategies, and observations on the SKIP Local Initiative Fund self-evaluation project from the perspectives of government purchaser and non-government organisation (NGO) providers, respectively.

The government purchaser (the Ministry of Social Development) believed that the SKIP Local Initiative Fund self-evaluation project has the potential to contribute to social change through creating relationships based on trust, focusing on capacity building, formulating self-evaluation questions and developing good practice. The NGO providers thought that participation in the self-evaluation project helped them to work in partnership with the Ministry of Social Development, develop clear goals and objectives, achieve several successful collaborative initiatives, evaluate outcomes and adjust them to community needs, and reflect on how the work done could have been improved.

Josie Roberts and Sally Boyd of the New Zealand Council for Educational Research (NZCER) facilitated a seminar on the evaluation of Secondary Futures, an autonomous organisation concerned with the future of secondary education in New Zealand. The facilitators discussed the issues they dealt with while evaluating the efficacy of Secondary Futures as an agent for social change to inform the development of the Secondary Futures / Hoenga Auaha Taiohi project in its second phase. The issues included:

• the “design challenges” for NZCER (small-scale evaluation, budget constraints, no set outcomes)
• the methods used (feedback from workshops, interviews)
• the findings concerning the activities of Secondary Futures as a catalyst for educational change and, by implication, social change (use of “people power”, organisation of grassroots support, formulation of strategies for long-term movement, promotion of paradigm shifts).

The seminar also explored the questions and challenges that NZCER wanted to address, including:

• Where should Secondary Futures position itself?
• What is the role of Secondary Futures in practice and policy?
• How does Secondary Futures propose to work towards supporting debate and visioning for the future of secondary education in New Zealand?
• Given that New Zealanders have started to become more “futures literate” in thinking about education, can we build momentum beyond “literacy” by building pathways, and creating a mandate, for change?

Social Change

One of the major themes that emerged in a number of keynote addresses and workshops was the need for evaluators to focus on the utilisation of their work by stakeholders. The ability of any evaluation to effect social change depends on the uptake and utilisation of evaluation findings by policymakers, programme designers and implementers. In this workshop, participants discussed their experiences of evaluations that had contributed to change. Pam Oliver offered a list of suggestions for increasing the social change potential of evaluation, including:

• focusing the evaluation on the intended uses of the evaluation findings
• designing evaluations to include opportunities for capacity building and reflective learning
• presenting findings at reporting workshops for all key stakeholders, and discussing the planned uses of the findings as part of the workshops
• disseminating findings widely (to participants, in conferences and in relevant journals)
• conducting a meta-evaluation to determine the value of the evaluation, and a follow-up at six months to ask how the evaluation has been used, what was useful and what was not.

Members of the Whāriki and SHORE evaluation group led a group discussion about the role of evaluators in promoting social change. In particular, the discussion dealt with the Māori approach to evaluation used by Whāriki, which emphasises the importance of conducting the evaluation within the cultural context of the subject programme. For Whāriki, this begins by using the Māori concept of “hikoi”, which means journey, to explain the collective endeavour of joint examination and learning that they as evaluators embark upon with their subject programme. The use of “hikoi” allows a myriad of touchstones for Māori culture to relate their values and goals to evaluation culture. In particular, the togetherness implicit in journey is of crucial importance for fostering trust and a focus on relationships that inform their work. Another important strand is humility – that while Whāriki provides some expertise in evaluation, they are not the experts, rather aiding the subjects to improve and develop their programme. Because of Whāriki’s declared role of being proponents of Māori development first and foremost, they tend to concentrate on formative evaluation contributing to Māori wellbeing and capacity for self-determination, as opposed to Western notions of evaluation for accountability. This line of reasoning opened up the issue of evaluators as agents for social change.

During the second half of the session the group sunk its collective teeth into discussing what it means to be agents for social change and if this is always a legitimate role for evaluation, or if there are situations where notions of objectivity have a valid role in evaluation. While almost all of those present would agree that programme evaluation serves social change, some of the participants diverged in that they saw evaluation taking, at times, a vantage of impartiality. Generally this difference amounted to nuance rather than attitude, and also tended to highlight differences between those working in government and evaluators working more at the community level.

Evaluation capacity and culture

Melissa Weenink and Kate McKegg facilitated a group discussion about the Ministry of Education’s evaluation strategy. One of the major themes to emerge was the importance of considering evaluation utilisation at an organisational level. Their study of evaluation utilisation with the Ministry of Education indicated that evaluation findings were most likely to be used when policy stakeholders were involved in the evaluation process from beginning to end. They found that where stakeholders had increased connection with the evaluation, it not only increased utilisation of the findings but also resulted in more positive perceptions of the quality and rigour of the evaluation. Building on findings from the evaluation utilisation survey, the presenters suggested that evaluation should be embedded into the processes of organisational policy development and implementation.

Jane Davidson (Davidson Consulting Limited) facilitated a seminar based on her book Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (published in 2004 by Sage Publications). Jane believes that the primary task of evaluation is to address how “good, valuable, or important” a project is in terms of meeting individuals’ needs. She argued that evaluation needs to blend descriptive facts (“what’s so”) with relevant values to draw evaluative conclusions (“so what”); is not supposed to be influenced by personal preferences or judgements or biases, although using “inner experiences” could be perfectly appropriate; and has to be discussed with and critiqued by stakeholders/audiences, and benchmarked against “other efforts or alternatives”. She maintained that in order to be an agent for social change, evaluation should be capable of identifying the criteria for measuring performance, and should be properly designed, culturally sensitive, compliant with legal/ethical requirements, transparent and credible.

Liz Smith, Sally Duckworth, Arti Badiani and Perise Iupeli (Litmus Research) focused on the role of evaluation in contributing to social change through facilitating effective engagement with a diverse range of stakeholders, such as end-users, service providers, and policymakers. The challenges confronted by evaluators in establishing meaningful contact with stakeholders (e.g. insufficiency of time and resources and lack of communication with hard-to-reach people and groups) were highlighted. Methods and techniques to make effective engagement possible were suggested, including honouring commitments and reaching out to socially isolated end-users. Audience participation was encouraged: as an example, the facilitators asked every participant to “tell a story” about what they did to achieve effective stakeholder engagement while managing one of their projects.

Partnerships

In the final keynote address, Dr Ricardo Millet (former Director of Evaluation at the Kellogg Foundation) spoke at length on building partnerships as a prerequisite of social change. He believes that partnerships between funders or “grantors” and providers or “grantees” who are “working in the field” could result in the development of relationships based on mutual trust and respect, the realisation of the promise of participatory policymaking, and the sharing of good practices. As well, based on his vast experience of social research and evaluation, Dr Millet argued cogently in favour of participatory evaluation. He maintained that participatory evaluation has the potential to contribute to effective deliberations among stakeholders, joint confrontation of challenges that impede effective programme implementation, the development of alternative plans of action, the welfare of the programme beneficiaries, and social change characterised by socioeconomic equity and justice.