Methodological Lessons Learned from the Process Evaluation of the Comprehensive Strategy For Serious, Violent, and Chronic Juvenile Offenders

Sanjeev Sridharan
Senior Evaluation Specialist
Westat
Rockville, Maryland

Elizabeth I. Lopez
Substance Abuse and Mental Health Services Administration
United States Department of Health and Human Services
Rockville, Maryland

Abstract

Five important methodological lessons learned from the national process evaluation of the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders, conducted in the United States, are discussed.

The Comprehensive Strategy is an example of a comprehensive community initiative (CCI), which is an initiative targeted at an entire community and intended to provide holistic, multifaceted responses to complex social problems.

The five lessons learned for future evaluations of CCIs are: (1) the need for explicating theories of change; (2) a process of clarifying stakeholder roles; (3) the importance of mechanisms for engagement of evaluator with sites; (4) the need to develop a stakeholder-driven anticipated timeline of change within communities; and (5) the value of empirically testing pathways of change in CCIs.

Introduction

In recent years, comprehensive community initiatives (CCIs) have increasingly been used as a means of addressing social problems in a number of sectors (Hebert and Anderson 1998, Kagan 1998, Kubisch et al. 1995, Milligan et al. 1998). In a recent issue of the Social Policy Journal of New Zealand, the relevance and promise of CCIs were explored in detail by Davies et al. (2002). As the popularity of CCIs increases, an exploration of methodological issues that are relevant for evaluations of CCIs is necessary. This paper will employ a large CCI implemented in the United States as a field example for methodological lessons learned.

CCIs target entire communities and are intended to provide holistic, multifaceted responses to complex social problems. CCIs have emerged in response to several trends, including:

• a recognition of the fragmentation of social services and the inappropriateness of responding to complex problems in a categorical manner
• funding requirements that encourage cost-effective prevention, reduction of expensive intervention measures and elimination of duplication in services
• a growing pragmatic and ideological support for the necessity of public-private partnerships and local action to solve social problems (Annie E. Casey Foundation 1995, Kubisch et al. 1997).

In the United States, CCIs have been implemented to address a wide variety of social problems. The Department of Justice has funded a number of CCIs to tackle a range of criminological problems. Examples of such initiatives include the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders (henceforth Comprehensive Strategy); Safe Futures; Safe Kids/Safe Streets; Safe Start; and the Greenbook Initiative. The Comprehensive Strategy, for example, targeted reduction in juvenile delinquency, Safe Start focused on minimising the negative impacts of children's exposure to violence in communities, and the Greenbook Initiative sought to build linkages between child-protective service organisations, domestic-violence agencies and the judicial organisations within a community.

There has been some discussion in the CCI literature on the types of evaluation methods that are useful for evaluating CCIs (Annie E. Casey Foundation 1997, Kubisch et al. 1995). These discussions have focused on issues of methodological design, as well as on the more challenging issues of the appropriate role of CCI evaluators (Brown 1995, 1998). In this paper we reflect on some of the methodological lessons learned from working on the national process evaluation of the Comprehensive Strategy. This discussion should be relevant to funders, programme staff and evaluators in New Zealand, who are considering implementing and evaluating CCIs.

The key conceptual problem that cuts across all of the lessons discussed here is on how the evaluation can positively influence the initiative. The lessons learned encompass a wide variety of issues including the relevance of a theory of evaluation influence (Henry and Mark 2003, Mark and Henry 2004), the importance of clarifying the roles of the various stakeholders involved in CCIs, and a discussion of "specialised" methods that may be more appropriate in evaluating CCIs. Examples of such methods include network analysis (Scott 2000) and concept mapping (Trochim 1999).

In this paper we first discuss some of the challenges involved in evaluating CCIs. We then briefly describe the background of the Comprehensive Strategy and the goals of the national process evaluation. Both the discussion on challenges of evaluating CCIs and the background of the Comprehensive Strategy should help the reader better understand the nature of CCIs. Finally, the methodological lessons learned in evaluating CCIs are discussed.

Challenges For Evaluating Comprehensive Community Initiatives

It is important to recognise that CCIs differ from conventional social programmes. An examination of the challenges involved in evaluating CCIs will help highlight the differences between CCIs and conventional programmes (Kubisch et al. 1995). Some of the challenges involved in evaluating CCIs include:

Operationally defining the initiatives

CCIs tend to be flexible (community-specific) and dynamic (evolving in response to lessons learned, as well as local needs and constraints). Given the potential variation in the scale, nature and sequence of the interventions, developing an evaluation framework that can be used to isolate programme effects is difficult.

Lack of appropriate control groups

While control groups can be created for individual programmes, control groups for comprehensive initiatives as a whole are very difficult to identify (Hollister and Hill 1995). The absence of appropriate control groups, coupled with the changing nature and variation in interventions across sites, makes causal attribution difficult. As an alternative, several researchers have proposed a focus on theory of change approaches (Connell and Kubisch 1998, Weiss 1995). A theory of change approach to CCI evaluation may be defined as "a systematic and cumulative study of the links between activities, outcomes and contexts of the initiative" (Connell and Kubisch 1998:16).

Horizontal and vertical complexity

CCIs are designed to promote horizontal collaboration between agencies and to affect outcomes at multiple (vertical) levels, including the individual, family, community and service systems. The evaluation challenge is to measure the synergistic benefits of collaboration between agencies, as well as to examine the relationship between and within each of the outcome levels. Measuring such collaborations and their respective benefits across multiple levels, and incorporating these findings into an operationalisable framework, is a complicated task.

Multiple outcomes and pathways of change

Since CCIs often represent diverse organisations with differing programme theories, there are multiple pathways by which interventions and processes can influence outcomes. Understanding and delineating these pathways and how they affect short-term, intermediate and long-term outcomes pose significant hurdles to evaluation. This is particularly important when there is diversity in the community contexts. The process of linking these varied contexts to individual outcomes in a measurable fashion is very complicated.

Different sites can also have different definitions of community

Within the same initiative, the community in question can range from small neighbourhood blocks to large communities. As an example, in the Comprehensive Strategy, communities included both a small zip code within Houston, Texas, as well as the entire county of San Diego, California (a county about the size of some states).

We do not attempt to "solve" the above methodological challenges in this paper. Instead, our approach through the thicket of methodological challenges is to focus on processes that will enhance the influence of the evaluation on stakeholders involved in the CCI. Recent work by evaluation theorists has stressed the importance of focusing on such a theory of influence (Kirkhart 2000, Henry and Mark 2003, Mark and Henry 2004) of evaluations. It is useful to think of the evaluation as contributing to a strategic conversation (Duignan 2002) between stakeholders. In our view, such a strategic conversation can be enhanced with a focus on the influence processes by which the evaluation can improve a CCI.

The Comprehensive Strategy

The Comprehensive Strategy provided a research-based framework for combating juvenile crime by targeting prevention efforts at youth who are at risk of delinquent behaviour, intervening in early delinquent behaviour, and responding effectively to youth who become involved in serious, violent and chronic offences. The Comprehensive Strategy's principal components of risk-focused delinquency prevention and graduated sanctions are intended to provide a "continuum of care" that both prevents and interrupts the progression of delinquent and criminal careers. The Comprehensive Strategy achieves this goal through a systematic community-level and research-based planning approach to crime and delinquency reduction.

The Comprehensive Strategy planning process involved several interrelated and ongoing steps:

• comprehensive training that mobilises the community, key leaders, and other stakeholders and familiarises them with the goals, principles and elements of the Comprehensive Strategy
• completion of community assessments (of risk factors, existing systems and resources, etc.)
• development of strategic plans (usually referred to as the five-year plan) that identify appropriate services, programmes and approaches to address the community's identified needs
• specifying programme mechanisms to coordinate and implement the plans
• implementing the plans, with ongoing monitoring of programme-specific and overall effectiveness at reducing risk factors and juvenile problem behaviours.

By undertaking and institutionalising this process - mobilise, assess, plan and implement - communities can adopt the Comprehensive Strategy framework.

A logic model that represents one version of the programme logic for the Comprehensive Strategy, and guided the process evaluation, is presented in Figure 1. The model describes the conceptual linkages between state-level and community-level contexts (social, economic and political conditions), the Comprehensive Strategy planning and implementation processes, and the outcomes at organisational, community, system and individual levels of the Comprehensive Strategy. Figure 1 also helps to describe the scope of the evaluation. The focus of the process evaluation was on structure and characteristics of the community planning efforts, the strategy of implementation and, to a limited extent, the immediate outcomes. The process evaluation did not focus on the intermediate and final outcomes described in Figure 1.

Process Evaluation Design

The national process evaluation incorporated a multi-method research design in which information from more than 40 sites was collected and analysed. The process evaluation spanned a period close to 30 months (June 1999 to December 2001). An important principle that guided the process evaluation was triangulation. Triangulation involved obtaining consistent results across multiple data collection and analytical methods. Table 1 describes the key goals and data collection activities for the national process evaluation. Given space considerations, our description of the evaluation is very brief. The details of the national process evaluation have been summarised in Sridharan et al. (2001).

The key goals for the national process evaluation (Sridharan and Gillespie 2004) included:

Documenting the Comprehensive Strategy planning and implementation processes

Given the wide variety of community contexts, an important goal of the evaluation was to document the planning and implementation activities in the sites.

Understanding strategic planning process

Given the importance of strategic planning in CCIs (Kubisch et al. 1997), we also examined the strategic planning processes in the Comprehensive Strategy. This was achieved through both intensive case studies and analysis of strategic plans from Comprehensive Strategy sites.

Figure 1 Illustrative Comprehensive Strategy Logic Model

Formative feedback to funder

We also focused on obtaining feedback through written surveys from the Comprehensive Strategy participants and trainers in improving the planning and implementation processes. We obtained feedback on a number of key issues, including potential improvements to training and technical assistance, the planning process, writing the five-year plan, and the role of the state and the funder.

Explicating the programme theory underlying the Comprehensive Strategy

The mechanisms of change are poorly understood in CCIs. The evaluation needs to help explicate the underlying programme theories. The explications of the programme theory occurred through two research activities: the interorganisation network surveys and a concept mapping process (Trochim et al. 1994) that mapped a stakeholder-driven anticipated timeline of change of outcomes relating to the Comprehensive Strategy (Sridharan and Zinzow 2003).

Table 1 Key Goals of Evaluation of the Comprehensive Strategy

Methodological Lessons Learned

We begin our discussion by briefly considering the relevance of a theory of evaluation influence. We then reflect on five methodological lessons learned that may be helpful in enhancing evaluation influence. We describe:

• the need for explicating theories of change
• a process of clarifying stakeholder roles
• mechanisms for the evaluator to engage with sites
• developing an anticipated timeline of change within communities
• empirically testing the pathways of change in CCIs.

We reiterate that our primary focus in this paper is on the methodological aspects of the lessons learned. More details on the substantive issues involving Comprehensive Strategy results can be obtained in Sridharan et al. (2001) and Sridharan and Gillespie (2004).

Towards a Theory of Evaluation Influence for CCIs

Theory of change approaches focus on pathways by which CCI processes affect short-term and long-term outcomes (Kubisch et al. 1995). Theory of change approaches can be economically represented by logic models that explicitly link programme processes (inputs) to short-term and long-term outcomes. We believe that most evaluations of CCIs can be significantly enhanced by using a process similar to a theory of change in describing pathways by which the evaluation can influence the initiative. Such an emphasis is consistent with recent focus by evaluation theorists on the importance of influence processes in evaluation (Kirkhart 2000, Mark and Henry 2004, Henry 2003, Henry and Mark 2003). These recent works have attempted to conceptualise evaluation activities as one of the links between social programmes and social betterment (Mark et al. 2000). As an example, consider Henry (2003:522):

These and similar evaluations invite us to reconsider how we think about the outcomes of evaluation and how we think about the attributes of evaluations that make them likely to enhance social betterment… The processes that influential evaluations can set off include changing attitudes or behaviors, persuading others, justifying policies and public expenditures, empowering change agents, and placing an item on the public agenda, among others.

Key questions to consider early in implementing a CCI (Sridharan 2003) include:

• Are there specific actions that funders, evaluators and other stakeholders can take to maximise the influence of the evaluation?
• What are the processes by which evaluation influences the initiative? Are there tools that can assist in understanding the timeline of "sequences" (Mark and Henry 2004) of influence processes?
• Are there structures of interaction between the client, programme staff and evaluator that promote such sequences of processes?

In our view, the process of building such a theory of influence should follow a process similar to developing a theory of change (Kubisch et al. 1995). It will involve active engagement of the stakeholders (including evaluators) and a process and accompanying structure to engage the stakeholders in an active, ongoing dialogue. Key themes that such a dialogue should emphasise include:

• clarifying the changing roles of stakeholders over the course of the initiative
• a stakeholder-driven anticipated timeline of change for the programme
• a discussion of methods and products that will help explicate the underlying theories of change.
  

The Need for Explicating Theories of Change

While an initial theory of change guided the implementation of CCIs, it is important to recognise most theories underlying CCIs are often incomplete or, perhaps, not well developed. Most initial logic models of CCIs are already quite complex. These logic models provide information on hypothesised linkages between key CCI processes and intermediate and final outcomes. Simply put, the logic model is a record of the anticipated theory of change underlying the CCI initiative within each of the communities. Rarely does a logic model contain a detailed discussion of context or of the embedded nature of the process of change (Pawson and Tilley 1997). An implemented CCI provides an opportunity to move from an initial theory of change to an emergent theory of change.

For example, while programme logic guided the implementation of the Comprehensive Strategy, the mechanisms that constituted the programme theory needed further explication. A description of the differences between the programme logic and programme theory is provided by Leeuw (2003:6):

Sometimes underlying assumptions are referred to as policy theories, but more often the terms program theory or program logic are used. An important difference exists between program theory and program logic. Program logic often specifies the inputs and components of a program, as well as short-term and long-term outcomes, along with the assumed linkages among these. However, program logic rarely outlines the underlying mechanisms that are presumed to be responsible for those linkages. In contrast, Rogers et al. (2000a:5, 2000b) see a program theory as an explicit theory or model of how a program causes the intended or observed outcomes.

This focus on explication is consistent with a growing emphasis in the programme evaluation literature on methods to "reconstruct" (Leeuw 2003:5) programme theories. This problem, referred to as "finding" (Cook 2000:29) the programme theory or the "inductive hunt" (Leeuw 2003:6), is driven both by the "lack of precision in most theories" (Bickman 2000:107) and the complexity of the change processes involved in CCIs.

The logic model provides one initial shared frame of reference of likely pathways of change in the community. Part of the evaluator's responsibility is to more fully document the actual processes of change within the community. Collaborative initiatives such as the Comprehensive Strategy typically involve considerable negotiations through multiple "roadblocks" with an attendant lack of movement towards goals. The emergent theory of change also needs to focus on such "roadblocks." Key questions for such an emergent theory of change include:

• What actions, events and, more generally, processes occurred in the community to overcome such roadblocks?
• How did the events, actions and contexts within each of the communities influence the short-term and long-term outcomes?
• What is an anticipated timeline of change of key outcomes (see discussion below)?

In our view, standard statistical techniques such as regression may not be too helpful in explicating such theories - methods such as regression are generally more useful in testing well-defined theories. We believe that there is a need for more specialised methodologies to help explicate the theories of change. Examples of such specialised methodologies implemented during the evaluation include methods of network analysis (Scott 2000, Gillespie and Murthy 1994) and concept mapping (Trochim et al. 1994, Trochim 1999).

As an example of such an explication process, network analysis methods were implemented to "unpack" the collaborative networks involved in planning and implementation of the Comprehensive Strategy processes. Taking a network perspective, we focus on both the structure of the overall collaborative network and the relationship of individual organisations with the overall network (Gillespie and Murthy 1994, Sridharan and Gillespie 2004). The key result from such a process of explication was that the one key organisational group that did not consistently participate in the Comprehensive Strategy was the local schools. Stakeholders mentioned difficulty in bringing school representatives to the table, especially leaders such as superintendents and principals.

Concept-mapping methodology (Trochim et al. 1994, Trochim 1999) can also help in explicating the theory of change by:

• empirically identifying the key dimensions involved in the planning and implementation phases of the Comprehensive Strategy
• helping prioritise outcomes that are most important to the stakeholders (this is turn provides a logical basis for the evaluator to decide which information is especially important to collect)
• developing an anticipated timeline of change that provides information on when changes in outcomes are expected to occur within the communities (we discuss the anticipated timeline of change in greater detail below).

A Process of Clarifying Stakeholder Roles

It is important to go through a formal process of establishing clarity of roles and expectations of all of the key stakeholders early in the evaluation. This is an especially important issue in the case of CCIs because of a potential conflict of roles. On the one hand, almost by definition, the community "owns" such an initiative. On the other hand, funders play a vital role in initiating (and perhaps sustaining in the short run) such projects. Further, funders are typically interested in generalising results from such initiatives to other communities based on tangible evidence of programmatic effectiveness.

Key questions for the funder in determining roles over time would include, What role would the funder play and for how long? If the funder plans to participate in these initiatives only for a short time, it would be helpful to have an exit strategy at the onset. If the length of the funders' participation depends on key performance measures of the initiative, it would be desirable to have upfront clarity on the performance measures that are being considered.

The issue of roles is even more challenging in the case of evaluators. The roles of the evaluator will typically change over the course of the initiative. In the initial stages, the role of the evaluator will be that of a "critical friend" (Casswell 2001) providing formative information to help with the CCI planning activities. As the initiative unfolds and develops, however, there is a need for the evaluator to answer the more summative question: Does the initiative work? Having an explicit, upfront discussion of roles is especially important because there is an often an understandable resistance in communities towards evaluations. From the perspective of the community members, the potential added value of the evaluation to the community is often unclear. An early discussion on the changing roles of evaluators can help build trust among all of the stakeholders involved in CCI.

Mechanisms for Engagement of Evaluator with Sites

Clearly established mechanisms are needed to encourage the active engagement of the evaluator with the sites. This is more challenging than it may initially appear. Local sites can be numerous and frequently the evaluator may not have the time and resources to work directly with all the sites. It is important that the flow of information between the sites and the evaluation is not simply unidirectional (often from the communities to the evaluator) but rather an engaged, flexible bi-directional process. A number of evaluations of CCIs involve a partnership between a national evaluator (often charged with a cross-site evaluation) and a local evaluator (often charged with the evaluation in a single locality). In our experience, there are very few examples of exemplary models of such partnerships.

Questions that need to be addressed in building a partnership between the local and national evaluators are:

• Who collects what data? This is an especially thorny question because data at multiple levels (including system, organisational, stakeholder, family and children) need to be collected, and collecting information at the individual level (especially family and children) can be both difficult and expensive.
• How does the national, cross-site evaluation strategy accommodate local concerns?
• How does the local evaluator benefit by collaborating with the national evaluator?
• How does the national evaluator benefit by collaborating with the local evaluator?

An explicit and early discussion on what the cross-site evaluation brings to both the local community and the local evaluator will help in building partnerships between the national and local evaluation teams. It is important that such discussions occur in an open, honest manner rather than being dictated in a top-down fashion.

Pay Attention to an Anticipated Timeline of Change within Communities

The Comprehensive Strategy, like other CCIs, was intended to influence outcomes on many levels: systems, communities, agencies, families, and children. An important question from both a programming and evaluation perspective is: When are such impacts likely? A number of researchers and practitioners have pointed out that CCIs such as the Comprehensive Strategy typically take a long time to affect children and families. Understanding the anticipated timeline of change of outcomes would help in understanding the process of systems change associated with CCIs. An awareness of the timeline of change for outcomes can assist with developing a data collection system to track progress associated with CCIs. As indicated earlier, the concept-mapping methodology was used to develop an anticipated timeline of change. Concept mapping used a stakeholder-driven process that combines three data-collection activities: brainstorming, sorting and ratings. Each of these activities is briefly described below.

In the brainstorming stage, an open-ended survey was mailed to site coordinators in all of the sites. Site coordinators at each of the sites led the planning and implementation processes. Fifteen site coordinators completed the open-ended survey, which was used to develop a list of outcomes that the Comprehensive Strategy could possibly affect. The survey identified 153 outcomes at this stage.

In the second stage, ratings and sorting surveys were completed with 20 respondents: 16 site coordinators and four staff members from the Office of Juvenile Justice and Delinquency Prevention (the funder of this initiative). Respondents were also asked to rate the relevance of each outcome on a Likert type scale ranging from 1 (not relevant at all) to 5 (extremely relevant). Respondents were also asked to identify the earliest anticipated impact of the Comprehensive Strategy. The question posed was:

From the start of the initial planning phase, how many years will (did) it take for the Comprehensive Strategy initiative to impact each of these outcomes? Please use the scale below. Circle "6 or More" for an outcome that will take 6 or more years, and "U" if Comprehensive Strategy is unlikely to impact this outcome.

Figure 2 describes the top 10 outcomes based on the mean relevance ratings aggregated across the 20 ratings surveys. The median time in years for the earliest impact is also represented. Figure 2 provides information on the outcomes that are especially important to the stakeholders. It also provides one logical basis for the evaluator to decide which information is especially important to collect.

Figure 2 Ten Most Relevant Outcome Measures from the Concept Mapping Surveys

One of the key results was that the earliest impact of the Comprehensive Strategy on individual-level outcomes is anticipated to take more than four years (see Figure 2). Furthermore, a number of system-level changes were anticipated to take at least three years. These results help emphasise the important fact that CCIs are complicated processes, and systems change takes time. An understanding of the anticipated timeline of change involved in reforming systems can have implications for allocating funding, supporting sustained implementation efforts, and developing data-collection systems.

Understand the Pathways of Change by which CCIs Affect Individual Stakeholders

Most CCIs have complex theories of change. Indeed, the notion of complex pathways of change is explicit in the logic model. However, it is important to recognise that the pathways in the logic model are the assumed, hypothesised processes of change. The evaluation provides an opportunity to examine whether the actual pathways of change follow the hypothesised pathways. Typically, CCIs attempt to link activities and contexts to specific outcomes across multiple levels, including systems, communities, agencies, families and children. In our example, using the written stakeholder surveys, we examined the pathways by which the Comprehensive Strategy planning processes affected the participating stakeholder (programme planners and programme staff).

Data for the stakeholder surveys were collected from stakeholders participating in Comprehensive Strategy planning and implementation activities in 23 Comprehensive Strategy communities. Approximately 10 to 15 key stakeholders within each of these communities were surveyed. The sample of respondents within each of the sites was developed in close cooperation with the site coordinator in each Comprehensive Strategy community. Of 337 surveys distributed, we received a total of 175, for a response rate of 52%.

One of the goals of the Comprehensive Strategy planning process was to elicit a sense of empowerment and ownership from the participating stakeholders. An underlying assumption of the programme theory was that individual actors who participated in the planning and implementation process would feel more empowered to make a difference in their communities. We explicated and tested a structural equation model (Joreskog and Sorbom 1993) by which the Comprehensive Strategy planning process could affect the individual stakeholder. We studied the effects of a number of factors on its impact on the individual stakeholder and the likelihood of future involvement of the stakeholder. Figure 3 describes the tested structural equation model. These factors include assessment of training and technical assistance, perceptions of community readiness, consistency of goals between the respondent's organisation and the Comprehensive Strategy, and intensity of participation (see Sridharan et al. 2001 for definitions of these terms). These findings helped provide an understanding of how contextual factors and the support of funders can influence stakeholder involvement over time.

Figure 3 Structural Equation Model to Test Pathways of Change to Affect the Individual Stakeholder

As an example of the utility of such an approach, we describe two results from the structural equation model (these results are described in greater detail in Sridharan et al. (2001) and were validated using other data collection methods): negative relationship between perceived community readiness and the assessment of training and technical assistance, and positive relationship between consistency of goals of the respondent's organisation with the Comprehensive Strategy and intensity of participation.

Individuals who perceived their communities to have higher levels of community readiness had a more negative assessment of the training and technical assistance process relative to stakeholders from communities with lower levels of perceived readiness. This result was validated with some of the interviews conducted during site visits. A number of stakeholders from sites that had an extensive history (of collaboration, that is, corresponding to communities with high levels of community readiness) stated that the training did not adequately take into account the pre-existing context of the community (e.g., history of prior initiatives, knowledge and expertise of the participants) in the training.

Participants who felt there was greater consistency between their organisation's mission and the mission of the Comprehensive Strategy were more likely to participate intensively in the process. This result provides some explanation for a consistent finding among the sites - that participation from the schools was quite limited in the Comprehensive Strategy process. This may be due to the fact that the mission of the schools is apparently different from that of the Comprehensive Strategy.

These results are only intended to illustrate an approach that seeks to confirm the pathways of change using empirical approaches. The broader point is that evaluations of CCIs need to balance theoretical approaches (represented in hypothesised theory of change) with empirical approaches. In our view, balancing theoretical with empirical approaches will result in a more refined theory of change for the initiative.

Conclusions

Evaluations of CCIs can contribute to a strategic conversation between stakeholders in the CCIs (Duignan 2002). In this paper, we have argued that for evaluations of CCIs to be truly strategic requires a process of clarifying roles of stakeholders, a closer attention to methods that can be helpful in explicating theories of change, developing structures of engagement of evaluator with sites, developing an anticipated timeline of change of outcomes to further clarify expectations of stakeholders, and empirically identifying pathways of change. While we do not purport these key elements of a CCI evaluation strategy to be exhaustive, we believe that straitjacket, "off-the-shelf" evaluation frameworks from more conventional programme evaluation strategies will likely be less than satisfactory for a value-added evaluation of CCIs. Methodological issues in CCIs are intricately connected to the issues of roles, structures of engagement and expectations of change and, as such, require careful, preliminary work early in the conceptualisation of the evaluation strategy. Moreover, ongoing dynamic and flexible strategies are necessary for CCI evaluation designs to maximally respond to the organic, process nature of these unique community programmes.

References

Annie E. Casey Foundation (1995) The Path of Most Resistance: Reflections on Lessons Learned from New Futures, Annie E. Casey, Baltimore, Maryland.

Annie E. Casey Foundation (1997) Evaluating Comprehensive Community Change, Report of the Annie E. Casey Foundation's March 1997 Research and Evaluation Conference, http://www.aecf.org/publications/evaluation/eval.pdf.

Bickman, L. (2000) "Summing up program theory" in P.J. Rogers, T.A. Hacsi, A. Petrosino and T.A. Huebner (eds.) Program Theory in Evaluation: Challenges and Opportunities, New Directions in Evaluation Volume 87, Jossey-Bass, San Francisco.

Brown, P. (1995) "The role of the evaluator in comprehensive community initiatives" in J.P. Connell, A.C. Kubisch, L.B. Schorr and C.H. Weiss (eds.) New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts, Aspen Institute, Washington D.C.

Brown, P. (1998) "Shaping the evaluator's role in a theory of change evaluation" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Casswell, S. (2001) "Community capacity building and social policy - What can be achieved?" Social Policy Journal of New Zealand, 17:22-35.

Connell, J.P. and A.C. Kubisch (1998) "Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Cook, T.D. (2000) "The false choice between theory-based evaluation and experimentation" in P.J. Rogers, T.A. Hacsi, A. Petrosino and T.A. Huebner (eds.) Program Theory in Evaluation: Challenges and Opportunities, New Directions in Evaluation Volume 87, Jossey-Bass, San Francisco.

Davies, E., B. Wood and R. Stephens (2002) "From rhetoric to action: A case for a comprehensive community-based initiative to improve developmental outcomes for disadvantaged children" Social Policy Journal of New Zealand, 19:28-47.

Duignan, P. (2002) "Building social policy evaluation capacity" Social Policy Journal of New Zealand, 19:179-194.

Gillespie, D.F. and S.A. Murthy (1994) "Cracks in a postdisaster service delivery network" American Journal of Community Psychology, 22:639-660.

Granger, R.C. (1998) "Establishing causality in evaluations of comprehensive community initiatives" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Hebert, S. and A. Anderson (1998) "Applying a theory of change approach to two national, multisite comprehensive community initiatives: Practitioner reflections" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Henry, G.T. (2000) "Why not use?" in V.J. Caracelli and H. Preskill (eds.) The Expanding Scope of Evaluation Use, New Directions for Evaluation, Jossey-Bass, San Francisco.

Henry, G.T. (2003). "Influential evaluations" American Journal of Evaluation, 24(4):515-524.

Henry, G.T. and M.M. Mark (2003). "Beyond use: Understanding evaluation's influence on attitudes and actions" American Journal of Evaluation, 24(3):293- 314.

Hollister, R.G. and J. Hill (1995). "Problems in the evaluation of community-wide initiatives" in J.P. Connell, A.C. Kubisch, L.B. Schorr and C.H. Weiss (eds.) New Approaches to Evaluating Community Initiatives, Volume 1: Concepts, Methods and Contexts, Aspen Institute, Washington D.C.

Howell, J. and J. Wilson (1993) Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders Program Summary, Office of Juvenile Justice and Delinquency Prevention, Washington D.C.

Joreskog, K.G. and D. Sorbom ( 1993) LISREL 8. Structural Equation Modeling with the SIMPLIS Command Language, Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Kagan, S.L. (1998) "Using a theory of change approach in a national evaluation of family support programs: Practitioner reflections" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Kirkhart, K. (2000) "Reconceptualizing evaluation use: An integrated theory of influence" in V. Caracelli and H. Preskill (eds.) The Expanding Scope of Evaluation Use, New Directions for Evaluation Volume 88, Jossey-Bass, San Francisco, CA.

Kubisch, A.C., P. Brown, R. Chaskin, J. Hirota, M. Joseph, H. Richman and M. Roberts (1997) Voices from the Field: Learning from Comprehensive Community Initiatives, Aspen Institute, Washington D.C.

Kubisch, A.C., K. Fulbright-Anderson and J.P. Connell (1998) "Evaluating community initiatives: A progress report" in J.P. Connell, A.C. Kubisch, L.B. Schorr and C.H. Weiss (eds.) New Approaches to Evaluating Community initiatives, Volume 1: Concepts, Methods and Contexts, Aspen Institute, Washington D.C.

Kubisch, A.C., C.H. Weiss, L.B. Schorr and J.P. Connell (1995) "Introduction" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement, and Analysis, Aspen Institute, Washington D.C.

Leeuw, F.L. (2003) "Reconstructing program theories: Methods available and problems to be solved" American Journal of Evaluation, 24(1):5-20.

Mark, M.M. and G.T. Henry (2004) "The mechanisms and outcomes of evaluation influence" Evaluation, 10(1):35-57.

Mark, M.M., G.T. Henry and G. Julnes (2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs, Jossey-Bass, San Francisco.

Milligan, S., C. Coulton, P. York and R. Register (1998) "Implementing a theory of change evaluation in the Cleveland community-building initiative: A case study" in K. Fulbright-Anderson, A.C. Kubisch and J.P. Connell (eds.) New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis, Aspen Institute, Washington D.C.

Pawson, R. and N. Tilley (1997) "An introduction to scientific realist evaluations" in E. Chelimsky and W.R. Shadish (eds.) Evaluation for the 21st Century: A Handbook, Sage, Thousand Oaks, California.

Rogers, P.J., T.A. Hacsi, A. Petrosino and T.A. Huebner (eds.) (2000b) Program Theory in Evaluation: Challenges and Opportunities, New Directions in Evaluation Volume 87, Jossey-Bass, San Francisco.

Rogers, P.J., T.A. Hacsi, A. Petrosino, and T.A. Huebner (eds.) (2000a) Program Theory in Evaluation: Practice, Promise and Problems, New Directions in Evaluation Volume 87, Jossey-Bass, San Francisco.

Scott, J. (2000) Social Network Analysis: A Handbook, Sage Publications, London.

Sridharan, S. and D. Gillespie (2004) "Sustaining formal problem-solving capacity in collaborative networks" Criminology and Public Policy, forthcoming.

Sridharan, S., H. Zinzow, A. Gray and E. Go (2001) National Process Evaluation of the Comprehensive Strategy for Serious, Violent and Chronic Juvenile Offenders, Caliber Associates, Fairfax, Virginia.

Sridharan, S. and H. Zinzow (2003) "The anticipated timeline-of-change as a frame of reference for evaluations of comprehensive community initiatives" unpublished.

Sridharan, Sanjeev (2003). "Introduction to special section on 'what is a useful evaluation?'" American Journal of Evaluation, 24(4):483-487.

Trochim, W., J. Cook, and R. Setze (1994). "Using concept mapping to develop a conceptual framework of staff's views of a supported employment program for persons with severe mental illness" Consulting and Clinical Psychology, 62(4):766-775.

Trochim, W. (1999) "The evaluator as cartographer: Technology for mapping where we're going and where we've been" presented to Conference of the Oregon Program Evaluators Network.

Weiss, C.H. (1995) "Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families" in J.P. Connell, A.C. Kubisch, L.B. Schorr and C.H. Weiss (eds.) New Approaches to Evaluating Community Initiatives, Volume 1: Concepts, Methods and Contexts, Aspen Institute, Washington D.C.

Wilson, J.J. and J.C. Howell (1995) "Comprehensive strategy for serious, violent and chronic juvenile offenders" in J.C. Howell, B. Krisberg, J.D. Hawkins and J.J. Wilson (eds.) Serious, Violent, and Chronic Juvenile Offenders: A Sourcebook, Sage, Thousand Oaks, California.