Discussion: Characteristics, Challenges, and Opportunities of Evidence-Based Design
Consider the following quotation: “Often times, potential users of research knowledge are unconnected to those who do the research, and consequently a huge gap ensues between research knowledge and practice behaviors” (Barwick, M., Boudell, K., Stasiulis, E., Ferguson, H., Blase, K., & Fixsen, D., 2005). Social workers must work to close the gap perceived by the authors of this quote.
In your previous research course, you addressed the concept of evidence-based practice. However, it is important not to fall into a habit of using the term “evidence-based practice” without a clear understanding of its meaning. In particular, it is important to understand what standards of evidence must exist to classify an intervention or a program as evidence based. In this assignment, you are to clarify your understanding of the nature of evidence-based practice and analyze the challenges and opportunities for implementing evidence-based practice in your current social work practice.
To prepare for this Discussion, read the Learning Resources that provide information about different aspects of the evidence-based practice concept. As you read, consider how evidence-based practice or evidence- based programs might be used in a social work agency where you work or where you had a practicum experience.
https://www.socialworkers.org/News/Research-Data/Social-Work-Policy-Research/Evidence-Based-Practice
By Day 3
Post a description of the distinguishing characteristics of evidenced-based practice. Then provide an evaluation of factors that might support or impede your efforts in adopting evidence-based practice or evidence-based programs.
In recent years, there has been increased pressure from funding agencies and
federal, state and local governments for greater effectiveness and accountability
of prevention and intervention programs. This rising demand for program
quality, and evidence of that quality, has fueled a growing interest in evidence‐
based programs (EBPs). However, there remains some confusion about what
constitutes an EBP, whether some EBPs are better than others, and the advantages
and disadvantages of implementing EBPs. In this Research to Practice brief, we
provide an overview of what it means for a program to be evidence‐based, discuss
the advantages and disadvantages of implementing EBPs, and point readers in the
direction of resources to help locate these programs and learn more about them.
What are evidence‐based programs? A growing body of research in the social and behavioral sciences has demonstrated that
certain approaches and strategies for working with youth and their families can positively
impact important social problems such as delinquency, teen pregnancy, substance abuse and
family violence. Many of these effective approaches and strategies have been packaged into
programs targeting outcomes specific to individuals, schools, families, and communities.
Those programs that have been found to be effective based on the results of rigorous evaluations
are often called “evidence‐based.”
WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES
Evidence‐based programs:
An overview
ISSUE #6, OCTOBER 2007
BY SIOBHAN M. COONEY, MARY HUSER,
STEPHEN SMALL, AND CAILIN O’CONNOR
University of Wisconsin–Madison and University of Wisconsin–Extension
Evidence‐based programs: An overview 2
What Works, Wisconsin – Research to Practice Series, #6
The importance of rigorous evaluation A rigorous evaluation typically involves either an
experimental design (like that used in randomized
controlled trials) or a quasi‐experimental design. In an
experimental design, people are randomly assigned to either
the treatment group, which participates in the program, or
the control group, which does not. After the program is
completed, the outcomes of these two groups are
compared. This type of research design helps ensure that
any observed differences in outcomes between the two
groups are the result of the program and not other factors.
Given that randomization is not always possible, a quasi‐
experimental design is sometimes used. In evaluations using
this design, the program participants are compared to a
group of people similar in many ways to the program
participants. However, because a quasi‐experimental
design does not randomly assign participants to program
and non‐program groups, it is not as strong a design as the
experimental approach. Because there may be unobserved
differences between the two groups of people who are
being compared, this design does not allow program
evaluators to conclude with the same certainty that the
program itself was responsible for the impacts observed.
Most programs have evaluation evidence from less
rigorous studies. Evaluations that do not include any type
of comparison group, for example, do not allow for any
conclusions to be made about whether the changes seen in
program participants are related to or caused by the
program. These studies sometimes show the promise of
positive results, but they do not allow the program to be
classified as evidence‐based. Programs with evidence from
less rigorous studies are often referred to as “promising”
programs.
An important element of EBPs is that they have
been evaluated rigorously in experimental or
quasi‐experimental studies (see box on this
page).
Not only are the results of these evaluations
important, but it is also essential that the
evaluations themselves have been subjected to
critical peer review. That is, experts in the field
– not just the people who developed and
evaluated the program – have examined the
evaluation’s methods and agreed with its
conclusions about the program’s effects. Thus,
EBPs often have evaluation findings published
in peer‐reviewed scientific journals.
When a program has sufficient peer‐
reviewed, empirical evidence for its
effectiveness, its developer will typi‐
cally submit it to certain federal
agencies and respected research
organizations for consideration. These
organizations “certify” or “endorse”
programs by including them in their
official lists of effective programs.
This lets others in the field know the
program meets certain standards of
effectiveness. (See Appendix A for
examples of these organizations.)
Simply put, a program is judged to be
evidence‐based if (a) evaluation re‐
search shows that the program pro‐
duces the expected positive results;
(b) the results can be attributed to the
program itself, rather than to other
extraneous factors or events; (c) the
evaluation is peer‐reviewed by
experts in the field; and (d) the
program is “endorsed” by a federal
agency or respected research
organization and included in their list
of effective programs.
Given this definition of an EBP, it is
important to distinguish the term
“evidence‐based” from “research‐
based.” Consider our earlier
description of how most, if not all,
EBPs were developed based on years
of scientific research on what program
components, such as content and
activities, are likely to work for youth
Evidence‐based programs: An overview 3
What Works, Wisconsin – Research to Practice Series, #6and families. Because EBPs contain program
components with solid empirical bases, they can
safely be called “research‐based” programs.
However, the reverse is not true. Not all, or
even the majority, of research‐based programs
fit the definition of an EBP. Just because a
program contains research‐based content or was
guided by research‐based information, doesn’t
mean it has been proven effective. Unless it also
has scientific evidence that it works, it is
incorrect to call it “evidence‐based.”
Are some evidence‐based
programs better than others? Programs that meet the definition of evidence‐
based are not all similarly effective or equally
likely to work in a given community.
For example, some EBPs have been evaluated
rigorously in several large‐scale evaluations that
follow participants for a long period of time.
Others have only undergone one or two less
rigorous evaluations (for example, those using
the quasi‐experimental design described on
page 2). Those programs that are shown to be
effective multiple times in experimental studies
are generally considered to be of a higher
standard.
Furthermore, many EBPs have been
successfully replicated and evaluated in a
variety of settings with a range of different
audiences. Others have only been evaluated
with a particular audience in a certain
geographical area, for example. When a
program has been shown to be effective in
different settings and with different audiences,
it is more likely that it will be effective when
implemented elsewhere.
Finally, EBPs can vary in the strength of their
effects. For example, one program may have
evidence that it reduces delinquent acts in its
participants by 10 percent over the subsequent
year, while another program has evidence of
reducing delinquency by 20 or 25 percent.
Generally, those programs that consistently pro‐
duce a greater effect than other programs are
thought to be better programs.
Thus, the level of evidence for effectiveness
varies across programs, and practitioners must
use a critical eye when judging where on the
continuum of effectiveness a program lies.
Advantages of evidence‐based
programs There are numerous merits to adopting and
implementing EBPs. First, utilizing an EBP in‐
creases the odds that the program will work as
intended and that the public good will be
enhanced. There is also greater efficiency in
using limited resources on what has been proven
to work as compared to what people think will
work or what has traditionally been done.
Instead of putting resources toward program
development, organizations can select from the
growing number of EBPs, which are not only
known to be effective but also often offer well‐
packaged program materials, staff training, and
technical assistance. Using EBPs where
appropriate can thus be viewed as a responsible
and thoughtful use of limited resources.
The proven effectiveness that underlies EBPs
can help secure resources and support from
funding agencies and other stakeholders, such
as policy makers, community leaders, and
members of the targeted population.
Increasingly, funders and policy makers are
recommending, if not requiring, that EBPs be
used to qualify for their financial support.
Additionally, the demonstrated effectiveness of
these programs can facilitate community buy‐in
Evidence‐based programs: An overview 4
What Works, Wisconsin – Research to Practice Series, #6and the recruitment and retention of program
participants.
A final benefit of EBPs is that they may have
cost‐benefit information available. This type of
information helps to convey the potential eco‐
nomic savings that can accrue when funds are
invested in a program. Cost‐benefit information
can be very influential in an era where
accountability and economic factors often drive
public policy and funding decisions.
Disadvantages of evidence‐based
programs Despite the numerous advantages of EBPs,
there are some limitations that are important to
consider. A major constraint is the financial
resources needed to adopt and implement them.
Most EBPs are developed, copyrighted, and
sold at rather substantial costs. Program
designers often require that organizations
purchase curricula and other specially
developed program materials, that staff attend
specialized training, and that program
facilitators hold certain degrees or certifications.
Furthermore, EBPs are often intended to be im‐
plemented exactly as designed, allowing little
room for local adaptation.
Finally, organizations sometimes find that there
are few or no EBPs that are both well‐suited to
meet the needs of targeted audiences and
appropriate for their organization and local
community setting. This situation is especially
common when it comes to the promotion of
positive outcomes rather than the prevention of
negative ones. Because the development of
many EBPs was sponsored by federal agencies
concerned with addressing specific problems,
such as substance abuse, mental illness,
violence, or delinquency, there currently exist
many more problem‐focused EBPs than ones
designed specifically to promote positive
developmental outcomes like school success or
social responsibility.
Where to find evidence‐based
programs Practitioners looking for an EBP to implement
in their community or learn more about these
programs will find the Internet to be their most
useful resource. As mentioned earlier, a number
of federal agencies and respected research
organizations “certify” or “endorse” programs
that meet the organizations’ specified standards
for effectiveness. Many of these agencies have
established on‐line registries, of lists of EBPs
that they have identified as effective. While
there are some differences in the standards used
by various organizations to assess whether a
program should be endorsed and thus included
on their registry, most share the primary criteria
regarding the need for strong empirical
evidence of program effectiveness.
Organizations that endorse EBPs typically limit
such endorsements, and thus their program
registry, to those programs that have shown an
impact on specific outcomes of interest to the
organization. For example, programs listed on
the Office of Juvenile Justice and Delinquency
Prevention’s Model Programs Guide have all
been shown to have an impact on juvenile
delinquency or well‐known precursors to
delinquency.
As previously mentioned, because the
development of many EBPs was funded by
federal agencies focused on specific problems,
most existing registries of EBPs are problem‐
oriented. Occasionally, EBPs are categorized
according to a strengths‐based orientation and
address outcomes related to positive youth
Evidence‐based programs: An overview 5
What Works, Wisconsin – Research to Practice Series, #6WHAT WORKS, WISCONSIN: RESEARCH TO PRACTICE SERIES
This is one of a series of Research to Practice briefs prepared by the What Works, Wisconsin team at the
University of Wisconsin–Madison, School of Human Ecology, and Cooperative Extension, University of
Wisconsin–Extension. All of the briefs can be downloaded from http://whatworks.uwex.edu.
This series expands upon ideas that are discussed in What Works, Wisconsin: What Science Tells Us about
Cost‐Effective Programs for Juvenile Delinquency Prevention, which is also available for download at the
web address above.
This publication may be cited without permission provided the source is identified as: Cooney, S.M.,
Huser, M., Small, S., & O’Connor, C. (2007). Evidence‐based programs: An overview. What Works,
Wisconsin Research to Practice Series, 6. Madison, WI: University of Wisconsin–Madison/Extension.
This project was supported, in part, by Grant Award No. JF‐04‐PO‐0025 awarded by the Wisconsin
Office of Justice Assistance through the Wisconsin Governor’s Juvenile Justice Commission with funds
from the Office of Juvenile Justice and Delinquency Prevention.
development, academic achievement, school
readiness and family strengthening.
While registries of EBPs are usually organized
around the particular outcomes the programs
have been found to impact, many programs,
especially those focused on primary prevention,
often have broader effects than this pattern
would suggest. Many EBPs have been found to
be effective for reducing multiple problems and
promoting a number of positive outcomes. For
example, a parenting program that successfully
promotes effective parenting practices may not
only reduce the likelihood of particular
problems such as drug abuse or aggression, but
may also promote a variety of positive
outcomes like academic success or stronger
parent‐child relationships. For this reason, you
will often see the same program appear on
multiple registries that focus on different types
of outcomes.
Now, more than ever, practitioners have
available to them a wealth of EBPs that build on
the best available research on what works.
Unfortunately, they are currently underused
and often not well‐understood. Although EBPs
do have some limitations, they can contribute to
a comprehensive approach to preventing a
range of social and health‐related problems and
enhancing the well‐being of individuals,
families and communities.
Evidence‐based programs: An overview – Appendix A 6
What Works, Wisconsin – Research to Practice Series, #6Appendix A
Evidence‐based program registries
The following websites contain registries, or lists of evidence‐based programs, that have met specific criteria
for effectiveness. Program registries are typically sponsored by federal agencies or other research organiza‐
tions that endorse programs at different rating levels based on evidence of effectiveness for certain participant
outcomes. The registries listed below cover a range of areas including substance abuse and violence preven‐
tion as well as the promotion of positive outcomes such as school success and emotional and social compe‐
tence. Generally, registries are designed to be used for finding programs for implementation. However,
registries can also be used to learn about evidence‐based programs that may serve as models as organizations
modify aspects of their own programs.
Best Practices Registry for Suicide Prevention
http://www.sprc.org/featured_resources/ebpp/index.asp
This registry, developed by the Suicide Prevention Resource Center (SPRC) and the American Foundation for
Suicide Prevention, includes two registries of evidence‐based programs. The first draws directly from a larger
registry‐ that of the Substance Abuse and Mental Health Administration’s (SAMHSA) National Registry of
Evidence‐Based Programs and Practices (NREPP). Users interested in finding out more about programs
drawn from this registry will be directed to the NREPP site. The second registry was developed by SPRC in
2005 and lists Effective and Promising evidence‐based programs for suicide prevention. This portion has fact
sheets in PDF format for users interested in learning more about the listed programs.
Center for the Study and Prevention of Violence, Blueprints for Violence Prevention
http://www.colorado.edu/cspv/blueprints/index.html
This research center site provides information on model programs in its “Blueprints” section. Programs that
meet a strict scientific standard of program effectiveness are listed. These model programs (Blueprints) have
demonstrated their effectiveness in reducing adolescent violent crime, aggression, delinquency, and sub‐
stance abuse. Other programs have been identified as promising programs. Endorsements are updated
regularly, with programs added to and excluded from the registry based on new evaluation findings.
The Collaborative for Academic, Social, and Emotional Learning (CASEL)
http://www.casel.org/programs/selecting.php
The Safe and Sound report developed at CASEL lists school‐based programs that research has indicated are
effective in promoting social and emotional learning in schools. This type of learning has been shown to con‐
tribute to positive youth development, academic achievement, healthy behaviors, and reductions in youth
problem behaviors. Ratings are given on specific criteria for all programs listed, with some designated
“Select” programs. This registry has not been updated since programs were reviewed in 2003.
Evidence‐based programs: An overview – Appendix A 7
What Works, Wisconsin – Research to Practice Series, #6Exemplary and Promising Safe, Disciplined and Drug‐Free Schools Programs
http://www.ed.gov/admins/lead/safety/exemplary01/index.html
The Department of Education and the Expert Panel on Safe, Disciplined and Drug‐Free Schools identified
nine exemplary and 33 promising programs for this 2001 report. The report, which can be found at this site,
provides descriptions and contact information for each program. The focus is on programs that can be imp‐
lemented in a school setting whether in the classroom, in extra‐curricular activities, or as after‐school pro‐
gramming.
Helping America’s Youth
http://guide.helpingamericasyouth.gov/programtool.cfm
This registry is sponsored by the White House and was developed with the help of several federal agencies.
Programs focus on a range of youth outcomes such as academic achievement, substance use, and delin‐
quency, and are categorized as Level 1, Level 2, or Level 3 according to their demonstrated effectiveness. The
registry can be searched with keywords or by risk or protective factor, and is updated regularly to incorpo‐
rate new evidence‐based programs.
Northeast Center for the Application of Prevention Technology (CAPT) Database of Prevention Programs
http://www.hhd.org/capt/search.asp
This site features a simple or advanced search function to find substance abuse and other types of prevention
programs and determine their effectiveness according to a variety of criteria. Also included is information
about the sources those agencies used for their evaluations, contact information, websites, domains, relevant
references, and a brief description of each program.
Office of Juvenile Justice and Delinquency Prevention (OJJDP) Model Programs Guide
http://www.dsgonline.com/mpg2.5/mpg_index.htm
The OJJDP Model Programs Guide is a user‐friendly, online portal to prevention and intervention programs
that address a range of issues across the juvenile justice spectrum. The Guide now profiles more than 200
programs – rated Exemplary, Effective, or Promising – and helps communities identify those that best suit
their needs. Users can search the Guide’s database by program category, target population, risk and protec‐
tive factors, effectiveness rating, and other parameters. This registry is continuously updated and contains
more programs than other well‐known registries, although many of these are Promising rather than Exem‐
plary or Effective.
Promising Practices Network on Children, Families and Communities
http://www.promisingpractices.net/programs.asp
A project of the RAND Corporation, the Promising Practices Network website contains a registry of Proven
and Promising prevention programs that research has shown to be effective for a variety of outcomes. These
programs are generally focused on children, adolescents, and families. The website provides a thorough
summary of each program and is updated regularly.
Evidence‐based programs: An overview – Appendix A 8
What Works, Wisconsin – Research to Practice Series, #6Social Programs that Work, Coalition for Evidenced‐Based Policy
http://www.evidencebasedprograms.org/
This site is not a registry in the conventional sense of the word in that it does not include and exclude pro‐
grams based on some criteria of effectiveness. Instead, it summarizes the findings from rigorous evaluations
of programs targeting issues such as employment, substance use, teen pregnancy, and education. Some of the
programs have substantial evidence of their effectiveness, while others have evaluation results suggesting
their ineffectiveness. Users are welcome to sign up for emails announcing when the site is updated.
Strengthening America’s Families: Effective Family Programs for Prevention of Delinquency
http://www.strengtheningfamilies.org/
This registry summarizes and rates family strengthening programs which have been proven to be effective.
Programs are designated as Exemplary I, Exemplary II, Model, or Promising based upon the degree, quality
and outcomes of research associated with them. A program matrix is also included, which can be helpful in
determining “at a glance” which programs may best meet community needs. This registry was last revised in
1999.
Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) National Registry of
Evidence‐Based Programs and Practices
http://nrepp.samhsa.gov/
The National Registry of Evidence‐based Programs and Practices (NREPP) is a searchable database with up‐
to‐date, reliable information on the scientific basis and practicality of interventions. Rather than categorizing
programs as Model, Effective, or Promising, NREPP rates the quality of the research findings separately for
each outcome that has been evaluated, as well as readiness for dissemination. Users can perform customized
searches to identify specific interventions based upon desired outcomes, target populations and settings.
Youth Violence: A Report of the Surgeon General
http://www.surgeongeneral.gov/library/youthviolence/chapter5/sec3.html
This report designates programs as Model or Promising and goes further than many other registries to also
include a “Does Not Work” category. General approaches and specific programs for the prevention of youth
violence are described at three levels of intervention: primary, secondary and tertiary. This report has not
been updated since its publication in 2001, but it is rare in that it discusses the cost‐effectiveness of the
programs.Evidence-based intervention and services for high-risk youth: a North American perspective on the challenges of integration for policy, practice and research
James K. Whittaker Charles O. Cressey Endowed Professor Emeritus, School of Social Work, University of Washington, Seattle,
Washington, USA
A B S T R AC T
This paper explores the cross-national challenges of integrating evidence-based interventions into existing services for high-resource- using children and youth. Using several North American model programme exemplars that have demonstrated efficacy, the paper explores multiple challenges confronting policy-makers, evaluation researchers and practitioners who seek to enhance outcomes for troubled children and youth and improve overall service effective- ness. The paper concludes with practical implications for youth and family professionals, researchers, service agencies and policy–makers, with particular emphasis on possibilities for cross-national collaboration.
Correspondence: James K. Whittaker, School of Social Work, University of Washington, 4101 Fifteenth Avenue NE, Seattle, WA 98105-6299, USA E-mail: jimw@u.washington.edu
Keywords: children in need (services for), evidence-based practice, research in practice, therapeutic social work
Accepted for publication: January 2009
I N T R O D U C T I O N
Across many national boundaries and within multiple service contexts – juvenile justice, child mental and child welfare – there is a growing concern about a proportionately small number of multiply challenged children and youth who consume a disproportionate share of service resources, professional time and public attention. While accurate, empirically validated popu- lation estimates and descriptions remain elusive. The consensus of many international youth and family researchers, including those reported by McAuley and Davis (2009) (UK), Pecora et al. (2009a) (US) and Egelund and Lausten (2009) (Denmark) in this present volume seems to be that some combination of externalizing, ‘acting-out’ behaviour, problems with substance abuse, identified and often untreated mental
health problems, experience with trauma and challeng- ing familial and neighbourhood factors are often, and in various combinations, manifest in the population of children and youth most challenging to serve. Many of these find their way into intensive out-of-home care services, and Thoburn (2007) provides a useful window into the out-of-home care status of children in 14 countries and offers useful observations on improvements in collecting administrative data for child and family services to inform both policy and practice. Others call for a critical re-examination of the present status of ‘placement’ as a central fulcrum in child and family services policy and practice (Whittaker & Maluccio 2002).
A sense of urgency is conveyed by the fact that many child and youth clients of ‘deep-end’, restrictive (out-of-home) services disproportionately represent underserved and often socially excluded families and communities of colour, and pose additional challenges in service planning around the cultural compatibility of proffered interventions (Blasé & Fixsen 2003; Barbarin et al. 2004; Miranda et al. 2005). Important work in this area includes ethnic and cultural
Author note: Portions of this paper in earlier form were presented by the author at the 8th and 10th annual EUSARF International Conferences at the University of Leuven, Belgium, 9–11 April 2003 and the University of Padova, Italy, 26–29 March 2008.
doi:10.1111/j.1365-2206.2009.00621.x
166 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
mailto:jimw@u.washington.eduvariations on known effective practices. Lau (2006), for example, offers a nuanced and sensitive treatment of actual and potential adaptations in existing parent training models. A basic concern with questions of equity and social justice, coupled with a growing scep- ticism about the efficacy of traditional residential, ‘place-based’ services, has heightened the search for more preventive, family- and community-based, cul- turally congruent service alternatives. All of this is set against a backdrop of concern about the state’s ability to provide effective parenting oversight and support for children in care, as well as those who remain with their families (Bullock et al. 2006). Fortunately, this search is occurring at a time when researchers in many countries are shedding light on mechanisms of risk and resilience (Sameroff & Gutman 2004), change processes involved in effective interventions (Biehal 2008) and the challenges faced by parents in multiply stressed environments (Ghate & Hazel 2002; Ghate et al. 2008) that are rich in their potential for contri- butions to intervention design and evidence-informed practice.
The primary purpose of this paper is to examine some of the challenges and opportunities in incorpo- rating evidence-based strategies and interventions into existing service systems to better meet the needs of high-resource-using children and youth. The growing corpus of empirical research on promising treatment strategies offers, if not clear-cut prescrip- tions, then rich implications for future policy initia- tives and service experiments.
Indeed, the pursuit of evidence-based practice, in its many forms, increasingly attracts the attention of those who plan, deliver and evaluate critical treatment and rehabilitative services for vulnerable children and their families across national boundaries and regions. While definitions of ‘evidence-based practice’ empha- size different dimensions of that construct, the common themes of bringing ‘science-to-service’, and its reciprocal ‘service-to-science’, are increasingly evident in the child, youth and family services systems in many European countries and North America, as well as elsewhere. Simultaneously, reform efforts in the USA and many European countries press for community-based, family-oriented, non-residential alternatives to traditional residential care and treat- ment programmes for acting-out children and youth with identified mental health problems (Chamberlain 2003; Weisz & Gray 2008). However, the impulse for service reform and the availability of at least some empirically validated model interventions do not of themselves constitute a sufficient basis for system
reform, but instead serve to illuminate some of the many fault lines that exist in the child and family services field:
• The continuing tensions between ‘front-end’, pre- ventive services and ‘deep-end’ highly intensive treatment services and the unhelpful dichotomies these tend to create and perpetuate
• The tensions between a widely shared desire to adopt more evidence-based practices and the genu- inely felt resistances to these, particularly when they are used in a rigid fashion that requires strict adher- ence to established protocols with little opportunity for experimentation, customization or practitioner discretion. For example, as one family support researcher recently observed, we need much more fine-grained analyses of the actual lived experience of client families with the services offered to them (S. P. Kemp 2008, personal communication). Such analyses will almost certainly involved a ‘mixed- methods’ approach using qualitative measures and methods to augment quantitative studies
• The tension, as manifested in North America and elsewhere between evidence-based and culturally competent practices, reflects, among other things, antagonism towards certain practice strategies based on perceptions of the under-representation of ethnic minorities in the study samples on which certain models have been validated As model programmes proliferate and are increas-
ingly removed from the particular political and cul- tural niches within which they were developed, we would do well to heed the cautions offered by Munro et al. (2005) that researchers, planners and youth and family practitioners are at a moment in time when cross-national perspectives are critical in helping iden- tify new ways of both framing problems and shaping service solutions. Cross-national dialogue can help in identifying different formats for collecting, analysing and utilizing routinely gathered client information, analysing subtle local adaptations of internationally recognized evidence-based services and examining the effects of differing policy contexts on service outcomes.
T H E Q U E S T F O R M O R E E F F E C T I V E I N T E R V E N T I O N S
For the remainder of this paper, I wish to do three things: (1) briefly identify where we are in our search for effective (evidence-based) interventions; (2) assess how we are doing in increasing their availability to high-resource-using troubled youth and their families;
Evidence-based intervention for high-risk youth J K Whittaker
167 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
and (3) identify some particular challenges faced by the individual practitioner, the social agency and the public policy context in furthering the shared goal of improving outcomes, and thus life prospects for troubled children. The author’s bias will soon be readily apparent. First, as one who has spent a lifetime trying to bring both the precision of research methods and the richness of research findings to the ‘shop floor’ of children’s agency practice, I am convinced that the evidence-based practice movement will not succeed until it is embraced by those closest to the children: the child and youth care workers, the social workers, teachers, family support workers and others who, with parents, toil on the front lines of helping. This is not in my view a one-way street – Science-to- Service – but presumes a vital feedback loop from Service-to-Science where the insights and hypotheses of those most directly involved in interventions (including parent and child consumers) inform and improve successive generations of applied research studies. Second, I readily acknowledge the North American bias apparent in many of my examples – I write of what I know best – while recognizing a deeply felt need in my country for European and other cross- national perspectives if we are ever to achieve success with our internal efforts at improving outcomes.
The search for evidence-based practices with chil- dren and families is now well underway on both sides of the Atlantic. Kazdin and Weisz (2003), Weisz (2004), Burns and Hoagwood (2002), Macdonald (2001), Pecora et al. (2009b) and McAuley et al. (2006) survey effective interventions in child welfare and child mental health services, as well as review current research on service populations that will inform the creation of novel interventions.
The simple, nominal definition of evidence-based practice offered by Professor Geraldine MacDonald of Queen’s University in Belfast provides a useful start- ing point:
Evidence-based practice indicates an approach to decision-
making which is transparent, accountable and based on careful
consideration of the most compelling evidence we have about
the effects of particular interventions on the welfare of indi-
viduals, groups and communities. (MacDonald 2001, p. xviii)
It is clear that debates about what constitutes the sufficiency and quality of evidence – where to set the bar for rigour, how to distinguish evidence-based vs. evidence-informed practice – continue apace both in academic and practitioner discourse even as the evidence-based practice movement as a whole contin- ues to raise its profile in policy and services. These
competing definitions and nuances are, in toto, a sign of health as they simply serve to underscore one or another aspect of what is emerging as a more fulsome understanding of what evidence-based practice con- sists of. These aspects include, but are not limited to:
• a dual focus on aetiology and outcomes • the incorporation of ethics and values as key com-
ponents
• the development of a collaborative process with affected client groups
• a commitment to transparency in processes and accountability Many practitioners and practice researchers have
participated in the work of international groups such as the Campbell and Cochrane Collaborations (Littell 2008) – originating in the health field – that attempt to sift, sort and categorize the state of the evidence around particular illnesses, socio-behavioural problems or social welfare concerns. Many have also experienced – closer to home – the increasing impact of national, state and regional initiatives designed to increase the content of proven, efficacious practices into child, youth and family service systems. Such initiatives typically use two strategies, often in combination:
Positive Reinforcement: e.g. ‘Laying Flowers Along Certain
Pathways’ by encouraging adoption of selected efficacious
model interventions. (One notes in passing that ‘efficacy’ of a
given intervention often increases in proportion to the dis-
tance from its country of origin!)
Coercion: e.g. Penalizing a programme, agency or practitioner
whose interventions do not reflect a sufficient quantity of
evidence-based practice according to an agreed-upon time
schedule. In the USA, this typically means that a practitioner
or service agency follows a prescribed protocol for interven-
tion or risks losing reimbursement for services rendered.
M OV I N G F R O M ‘ E F F I C AC Y- T O - E F F E C T I V E N E S S ’
In the USA at the moment, there is growing respect for the complexities involved in moving from pilot demonstrations of effective child, youth and family interventions to broad-scale application: i.e. moving from ‘efficacy’ to ‘effectiveness’ (Jensen et al. 2005; Weisz & Gray 2008). What these terms signify are: 1. That individual investigators can demonstrate sig- nificant results for novel treatments over standard (or traditional) services through carefully controlled, rig- orously conducted studies often including random- ized controlled trials: the ‘gold standard’ of clinical research. That is, they can demonstrate efficacy.
Evidence-based intervention for high-risk youth J K Whittaker168 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
2. Yet, these impressive results do not, on close examination, appear to influence what might be thought of as routine, day-to-day practice as con- ducted in more familiar agency settings. Thus, the evidence-based practice movement, while demon- strating efficacy, cannot as yet demonstrate overall effectiveness.
What explains this disconnect? Lisbeth Schorr, an astute analyst of child and family services innovation, sums it up succinctly: ‘Successful programs’, she says, ‘do not contain the seeds of their own replication’ (Schorr 1993, quoted in Fixsen et al. 2005).
Thus, if we are truly interested in effectiveness – i.e. achieving wide-scale adoption of proven efficacious interventions, we need to look beyond efficacy studies: (1) to those contextual elements that influence prac- tice decisions and client outcomes (Kemp et al. 1997); and (2) to a different kind of research undertaking that focuses directly on the processes involved in suc- cessful adoption of proven efficacious interventions (Weisz & Gray 2008).
John Weisz, one of the nation’s leading research analysts in child mental health and a professor of psychology at Harvard University as well as President of the Judge Baker Children’s Center in Boston, points the way forward on what is needed to ultimately resolve the efficacy/effectiveness challenge:
A very important focus for the next stage of research on
interventions for children will be the effective implementation
of evidence-based practices by practitioners in service settings.
This will require an active collaboration between the research-
ers who develop and test interventions and the clinical, child
welfare, and education professionals who serve children and
families. (J.R. Weisz 2008, personal communication)
E X P L O R I N G T H E L A N D S C A P E O F E V I D E N C E - B A S E D S E R V I C E S F O R H I G H - R I S K YO U T H
Let us proceed, then, by exploring the context within which evidence-based services are nested. Here, we find some common and proximate elements familiar to all who labour in the child and family services field, as well as a few more distal forces that, nonetheless, have a potential for considerable impact on the identification, validation and eventual integration of evidence-based practices. I will refer, briefly, to more or less typical examples from within the US context.
Model intervention programmes
For purposes of illustration, I offer three interventions that have received considerable attention in children’s
mental health services in the USA, and which have been the objects of numerous community replications and research study both in North America and else- where (Whittaker 2005). These include:
• Multisystemic Therapy (MST), developed principally by Dr Scott Henggeler, a psychologist now at the Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina (Henggeler et al. 1998; Schoenwald & Rowland 2002; Henggeler & Lee 2003). http://www.mstservices.com
• Treatment Foster Care (MTFC), developed in several clinical/research teams in the USA and represented here by the model (Multi-dimensional Treatment Foster Care) principally developed by Dr Patricia Chamberlain and colleagues at the Oregon Social Learning Center – a highly influential applied behaviour analysis developmental research centre – one of whose founding members is Dr Gerald Patterson (Chamberlain & Reid 1998; Chamberlain 2002, 2003). http://www.MTFC.com
• Wraparound Treatment, a novel, team-oriented, community-centred intervention developed by a variety of individuals including the late Dr John Burchard, formerly Professor of Clinical Psychology at the University of Vermont, John Van Den Berg, Carl Dennis and others beginning in the early 1980s (Burns & Goldman 1999; Burchard et al. 2002). http://www.rtc.pdx.edu/ PDF/PhaseActivWAProcess [While space does not permit in depth analysis here,
the interested reader is directed to the previously cited references, as well as to the web sites for each of these three models that include multiple references to com- pleted and in-progress research and demonstration efforts, as well as specifics on programme principles and components. A variation of the of the MTFC model designed for younger children in regular foster care is described in this present volume by Price et al. (2009)].
These three interventions are specifically designed to provide alternative pathways for children who otherwise would be headed into more costly and restrictive residential provision. Dr Barbara Burns, Professor of Psychology at Duke University in North Carolina and a principal author of the children’s mental health section of our latest Surgeon General’s Report on Mental Health (US Department of Health and Human Services 1999) provides a succinct ratio- nale for why this is warranted:
The most critical question for the future is, what will it take
to convince payers, public and private, to support the
Evidence-based intervention for high-risk youth J K Whittaker169 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
http://www.mstservices.com http://www.MTFC.com http://www.rtc.pdx.eduinterventions that are backed up by evidence about improved
outcomes? Assuming that the pool of dollars available for
mental health treatment will not increase, it will be necessary
to shift resources away from institutional care (which lacks
evidence of effectiveness) toward community alternatives.
This will require a reduction in funds allocated to institu-
tional care, where a significant portion of the child mental
health money is still being spent. (Burns & Hoagwood 2002,
p. 13)
While reviews of residential care in both the UK (Sinclair 2006) and the USA (Whittaker 2006) confirm a move away from residential services, recent comparative international contributions have urged critical re-examination of the multiple varieties of residential service (Courtney & Ivaniec 2009) to meet the needs of at least some high-resource- using youth. In part, this sentiment reflects the fact that theory and model development, particularly in the arena of intensive residential services has lan- guished as development of comparable family- centred services has flourished. Some have urged the development of a conceptual schema for intensive services – e.g. the ‘prosthetic environment’ – which transects more traditional residential, family and community boundaries is strengths-oriented and incorporates educational, socialization and family support services along with intensive treatment (Whittaker 2005).
In focusing here on a few programme models spe- cifically designed to serve as alternatives to residen- tial care and treatment, and other forms of intensive out-of-home service, one must acknowledge omis- sion of a great deal of promising, empirically based work that is presently being done with a wide range of family-, school- and community-centred interven- tions that is both more preventive in its focus and appropriate for a much wider population of children and families than space allows us to examine here. See, for example, Carolyn Webster Stratton’s Incred- ible Years Program (Beauchaine et al. 2005) and the work of many others whose contributions in such areas as family support illuminates a segment of ser- vices more preventive in focus (Kemp et al. 2005; Lightburn & Sessions 2006) and the contribution of Jackson et al. (2009).
What, then, are the similarities and differences of these three promising interventions? A recent review (Burns & Hoagwood 2002) yields the following: 1. All three interventions adhere to ‘systems of care’ values: The ‘systems of care’ framework derives from both our National Institute of Mental Health and
private foundation initiatives in the 1980s, and is defined as:
A comprehensive spectrum of mental health and other neces-
sary services which are organized into a coordinated network
to meet the multiple and changing needs of children and
adolescents with severe emotional disturbances and their
families. (Stroul & Friedman 1986, p. xx)
The system of care thus defined is based on three main ele-
ments. First, the mental health service system efforts are
driven by the needs and preferences of the child & family and
are addressed by a strengths-based approach. Second, the
locus and management of services occur within a multi-
agency collaborative environment grounded in a strong com-
munity base. Third, the services offered, the agencies
participating and programs generated are responsive to cul-
tural context and characteristics. [Though, as noted, this
remains a contested area with respect to some communities of
color.] (Burns & Hoagwood 2002, p. 19)
2. All three interventions are delivered in a commu- nity – home, school, neighbourhood – context as opposed to an office 3. All have operated in multiple service sectors: mental health, juvenile justice, child welfare 4. All were developed and evaluated in ‘real world’ community settings, thus enhancing external validity 5. All show preference for the model treatment con- dition in multiple randomized controlled trials 6. All lay claim to being less expensive to provide than institutional care (Burns & Hoagwood 2002, p. 7).
Differences of course exist. For example, both MST and MTFC possess a higher degree of specificity with respect to intervention components than does wrap- around. As of this writing, MST has perhaps the strongest evidentiary base, particularly in clinical trials showing positive effects, though some recent reviews, including one by Prof. Julia Littell of Bryn Mawr University in Pennsylvania conducted for the Camp- bell Collaboration, have raised critical questions about the evidence base offered in support of MST (Littell 2005, 2008). Finally, from a staffing perspective, MST appears to make higher use of master’s-level-trained professionals in service delivery than either MTFC or wraparound.
To these three model programmes, we must of course add numerous other evidence-based treatment techniques targeted to specific conditions and prob- lems, as reflected in recent reviews by Kazdin and Weisz (2003), Weisz (2004) and Chorpita et al. (2007). These model intervention programmes do not of course exist in a vacuum, but both influence and are influenced by a host of other elements in a typical state or regional context in the USA.
Evidence-based intervention for high-risk youth J K Whittaker170 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
P U B L I C , V O L U N TA R Y A N D P R O P R I E TA R Y S E R V I C E P R OV I D E R S
Model programmes such as MST, MTFC and wrap- around are typically adopted by some segment of the mixed system of service agencies (Public/Voluntary/ Proprietary) that make up the delivery system in a given state, county or municipality. Public service pro- viders are typically service funders as well, creating in the view of some voluntary agencies an unequal influ- ence in terms of what particular models are selected for adoption, as well as on the masking of true admin- istrative costs of programme implementation, given the public sector’s economies of scale and presumed ability to mask start-up costs. Given the wide varia- tions in state and county service systems within the USA, there are some anecdotal reports of the ten- dency of certain model programmes to bend and shape themselves into a widely varying array of funding arrangements (referred to as ‘pretzelling’) in order to gain a foothold and a leverage in a given public system (K. Blasé 2007, personal communica- tion) with the result that local service providers may be held to similar outcome and process standards while enjoying widely varying reimbursements to support their efforts.
N AT I O N A L , R E G I O N A L A N D L O C A L R E S E A R C H C E N T E R S A N D R E S O U R C E N E T W O R K S
In addition to evidence-based programme models that typically have their own internal capacity for pro- gramme development, marketing, training, evaluation and dissemination, a wide variety of university and institute-based resource networks and research centres play an increasingly important role in the promotion of evidence-based programmes and practices. For example, the National Implementation Research Network (NIRN) was begun at the University of South Florida as part of a larger effort to bring science-based information to the forefront of child mental health practice. Recently relocated to the University of North Carolina, NIRN has done significant work in documenting national, state and regional capacity to support model programme development, and has provided consultation to individual states and organi- zations on effective strategies for integrating evidence- based practices into the fabric of existing services (Fixsen et al. 2005). For more information, see: http:// www.fpg.unc.edu/~NIRN/. The California Evidence- Based Clearinghouse for Child Welfare Practice is
funded by the California Department of Social Ser- vices, Office of Child Abuse Prevention and guided by a state advisory committee and a National Scientific Panel. The Clearinghouse provides guidance on selected evidence-based practices in simple straightfor- ward formats, reducing the consumer’s need to conduct literature searches, review extensive literature or understand and critique research methodology (http://www.cachildwelfareclearinghouse.org/). The Clearinghouse has developed a six-tiered schema for sorting out promising programmes ranging from ‘Well-Supported – Effective Practice’ to ‘Concerning Practice’ (e.g. shows negative effects on clients and/or potential for harm).
A legislatively generated state institute, the Washington State Institute on Public Policy (WSIP) was created by the Washington state legislature to conduct cost/benefit and a range of other studies on a variety of classes of intervention, including child welfare and early intervention (http://www.wsipp. wa.gov/board.asp). Its generally thorough and well- executed analyses have achieved wide dissemination beyond the region and are frequently cited by model programme developers as confirmation of their effec- tiveness. Methodological concerns have recently been raised about the general quality of intervention research reviews (Littell 2005, 2008), including those generated by WSIP, and within local practice commu- nities, one hears anecdotally some concerns about the potential for overly concrete inferences by legislative bodies and funding sources whose attention may extend only to the executive summary section of detailed reviews of model programmes and not to the caveats and nuances contained in their appendices and footnotes.
Beyond these particular exemplars, there are a wide variety of government-, university- and institute- based research centres and clearinghouses devoted to the identification, review, evaluation and promotion of evidence-based practices. Such centres are not typically coordinated, resulting oftentimes in an over- load of information for busy practitioners desirous of identifying the most appropriate interventions for troubled youth and their families. The problem is intensified as estimates place the number of docu- mented treatments for children and adolescents in excess of 500 (Kazdin 2000). Here, the work of Dr Bruce Chorpita at the University of Hawaii offers at least a partial solution. For a number of years, Chor- pita’s research team has been refining a ‘common elements’ approach to identified evidence-based treatments and then matching these with identified
Evidence-based intervention for high-risk youth J K Whittaker171 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
http://www.fpg.unc.edu/~NIRN http://www.fpg.unc.edu/~NIRN http://www.cachildwelfareclearinghouse.org http://www.wsipp.wa.gov/board.asp http://www.wsipp.wa.gov/board.aspproblem clusters and characteristics of youth and families in service systems. The model’s particular focus on the practitioner’s adoption of discrete strat- egies, as opposed to whole-cloth approaches, is directly addressed to one of the major identified bar- riers to the implementation of evidence-based prac- tice: the resistance to treatment manuals (Chorpita et al. 2007). In a related area, the empirical research of Professor Charles Glisson of the University of Ten- nessee and colleagues sheds important light on the organizational factors that may impede or enhance the uptake of evidence-based practices in service set- tings: e.g. organizational structure, organizational culture and organizational climate (Glisson et al. 2008).
J U D I C I A L A N D L E G I S L AT I V E I N I T I AT I V E S
Vocal community advocacy calling attention to service inadequacies and lacunae – for example, failure to meet the mental health needs of children in the state foster care system or excessive numbers of placement changes – frequently end up in the court system. The resultant settlements, or ‘consent decrees’, can exert considerable direct and indirect pressure on the service system to adopt particular models of evidence- based practice as a remedy to the perceived problem. In addition, within an individual state or jurisdiction, there are not infrequently legislative initiatives designed to promote certain evidence-based practices, as well as initiatives generated from within the public service agency itself. Taken together with the already identified promotional efforts of model programme developers, sometimes augmented by the largesse of voluntary foundations that seek to promote particular strategies for service improvement, the resulting pres- sure for individual practitioners and voluntary service agencies to follow certain prescribed pathways to practice can be intense.
E V I D E N C E - B A S E D P R AC T I C E : M E E T I N G T H E C H A L L E N G E O F I M P L E M E N TAT I O N
For each of the features of the evidence-based practice landscape – model programmes, public and voluntary service providers, individual youth and family practi- tioners, research centres and clearinghouses, legisla- tive and judicial bodies and client communities – there are challenges to achieving the generally agreed-upon goal of improving outcomes for high-resource-using youth and their families through the adoption of proven, efficacious practices. While these challenges
vary depending on the point one occupies in the overall landscape of evidence-based practice, there appears to be a growing consensus in the USA for a far more intensive focus on what some have termed ‘implementation science’:
. . . the scientific study of methods to promote the systematic
uptake of clinical research findings and other evidence-based
practices into routine practice. (Implementation Science: UK:
on-line journal)
Thus, while different in their focus: (1) the previ- ously cited efforts of NIRN to identify effective path- ways for the integration of evidence-based practices into existing service systems (Fixsen et al. 2005); (2) the plea from research scholars like Julia Littell (2008) and others to bring more rigour, precision and sys- tematization to the scientific review processes for evidence-based approaches; and (3) the numerous contributions of senior research analysts like John Weisz and others (Weisz & Gray 2008), directed towards identifying pathways for bringing practitio- ners and researchers into a closer working relation- ship, are best viewed as part of a unified effort. There is, I believe, a growing awareness that integration of proven efficacious practices in youth and family work will happen only when there is a fully functioning infrastructure to support desired changes and various individual actors see their ‘part’ in relational to the ‘whole’.
Thus, for model programme developers, there is the critical task of identifying what are the active ingredi- ents in their interventions. What are the necessary and what are the sufficient components in a service unit of MST, MTFC or wraparound? Despite the fact that raising the question of ‘active ingredients’ leads one, ineluctably, to what noted child psychiatry researcher Peter Jensen calls ‘the soft underbelly’ of evidence- based treatment, it is an area of critical importance for future research (Jensen et al. 2005). The costing and ‘scaling-up’ implications alone of adding even a modest increment of evidence-based practice to exist- ing services warrants seeking answers to the question: ‘How much of what is enough’? There has been an understandable resistance on the part of many model programme developers to disaggregate their interven- tions for fear of compromising treatment integrity, and thus weakening outcomes. That said, it is heart- ening to note the flexibility of some models to cus- tomize their interventions to fit the needs of particular service populations and environmental niches. The previously cited modification of the MTFC pro- gramme reported elsewhere in this volume offers one
Evidence-based intervention for high-risk youth J K Whittaker172 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
such excellent example (Price et al. 2009). We must find ways to hold model programme developers harm- less for their results if they are willing to experiment with modifications of their ‘packages of service’ to address particular needs.
From the perspective of the individual service agency – whether it is a voluntary body, local authority or a large, public bureaucracy – a key question vis-à- vis the adoption of exemplary evidence-based pro- gramme models concerns the basic strategy for implementation: is it to be additive or integrative?
The additive approach that appears to prevail in many sectors of service in the present US context means that service agencies adopt one model pro- gramme at a time, adhering strictly to the interven- tion, assessment, training and evaluation protocols of the developer. This is meant to insure model fidelity and programme (treatment) integrity and to prevent what has been called ‘program contamination’. The result, in the author’s view, can lead to an encapsula- tion of discrete programmes – each with its identified staff and protocols for assessment and intervention – within a single agency structure. This results in fewer opportunities for cross-fertilization (e.g. common, or cross-training) and is silent on the preferred order of implementation: e.g. does it make a difference which model is adopted first? Moreover, the administrative complexities involved in managing multiple discrete programmes in a single agency can be considerable, particularly in smaller units with limited supervisory resources.
An alternate, or what might be termed an integra- tive approach, would seek to identify common ele- ments across successful model programmes and train towards those. The previously cited work of Chorpita et al. (2007) provides a potentially valuable founda- tion to such an approach. A slight variation on the integrative approach would be to identify a common platform of foundational knowledge and skill – e.g. around client engagement – or ‘therapeutic alliance’, the present legatee of the old concept of ‘relation- ship’ (Rauktis et al. 2005), and first establish that core competency with all staff before moving to incorporate the specific strategies and techniques contained within successful model programmes. At present, the enthusiasm for what might be called the ‘intervention-du-jour’ seems to suggest a continua- tion of the additive, seriatim, approach at the expense of the integrative.
Behind the specific issue of the preferred method for adopting efficacious model interventions lies the broader question of the service agency’s capacity to
integrate, utilize and generate practice-focused research. Weisz and others have proposed models for closer integration of research and practice (Hoagwood et al. 2002; Weisz & Gray 2008) within the service agency, but at the moment these are not widely in evidence. Whittaker et al. (2006) offer a five-stage model template for integrating evidence-based prac- tice in a child mental health agency, including logic modelling of existing programmes as a means of developing a common language of service, including implicit theories of change, selected evaluation activi- ties, strategic researcher–agency staff partnerships and benchmarking against practice models of national sig- nificance. A barrier to building research capacity in existing agencies is that present contracts are typically tied to designated services, not to building an infra- structure supportive of research.
Challenges for the service agency include data management where the adoption of electronic records lags in certain sectors and where many agencies lack the capacity to systematically analyse routinely gath- ered data at either the case or the aggregate level. As noted, the proliferation of assessment and evaluation measures – often tied to specific programmes – adds complexity to the data management needs of the service agency. In the critical arena of supervision, the question arises of the adequacy of a single supervisor to provide oversight and support to a staff operating in widely disparate intervention models with their differing change theories, assessment protocols, outcome measures and time frames. In the related arena of training, similar problems can be found, including almost exclusive reliance on the use of external (and often expensive) consultants during the start-up phase of a model intervention with unclear plans for transition of oversight to internal agency staff. Moreover, the determinants for training foci in some agencies remain strongly with worker interests and are not necessarily related to client characteristics. Of particular concern among many smaller, voluntary service settings in the USA is the factor of agency history. Many such agencies were residential in their origins, typically following a pathway from orphanage to treatment setting. Thus, boards of governors and major donors may be more oriented to place-based services and ‘bricks & mortar’ than to community-based programme alter- natives. A quote from a senior head in one such agency captures the tension for those in leadership: ‘How do I insure that my program plan for the agency is in synchrony with my business plan?’ (K. Scott 2002, personal communication).
Evidence-based intervention for high-risk youth J K Whittaker173 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
S U M M A R Y A N D TA K E - AWAY M E S S AG E S
The growing corpus of research on evidence-based approaches to work with troubled youth and their families offers both hope and challenge. In the area of alternatives to residential services for high-resource- using youth, model programmes like MST, MTFC and wraparound appear to hold much promise, but their full-scale adoption into existing service systems will require addressing a series of complex implemen- tation challenges. Public sector children’s services often work with families who are challenging to engage and for whom permanency and continued child safety remain as core service objectives. The sensitive application of evidence-based programmes and practices into the real world of contemporary child and family practice must necessarily involve parents, social workers, model developers and researchers in bidirectional communication. All of these efforts will benefit greatly from sustained and multilevel cross-national collaboration. Of the several US originated programme models identified in this brief review, virtually all are intruding to some degree on the work-plans of service planners, evaluation researchers, supervisors and practitioners in Europe, Australia and elsewhere. Similarly, interventions or intervention components as varied as ‘Patch’ (geo- graphically centred, generalist services) (Adams & Krauth 1995) and ‘Family Group Conferencing’ (Pennell & Anderson 2005) have come to the USA from the UK, New Zealand and elsewhere in recent years. Since all of these ‘imports’ will likely undergo modification and appear again as ‘exports’, it behoves staff at all of the above levels to carefully track how these novel interventions are being incorporated into widely differing political, geographic, cultural and organizational contexts. Fortunately, the wide avail- ability of instantaneous, direct, point-to-point elec- tronic communication and the increasing prominence of cross-national journals, networks and conferences make such communication more possible than ever. From a research perspective, the widely varying envi- ronments into which model programmes are being introduced hold the distinct possibility for compara- tive research, including natural experiments.
For youth and family practitioners, service agencies and researchers and policy-makers, some concrete take-aways include: 1. For practitioners
• Challenge the ‘conventional wisdom’ of practice wherever it resides – including in your own per- sonal theories of change: for example, ‘insight is a
requisite for behaviour change’; ‘longer service produces better outcomes’. • Seek out and read one up-to-date review of inter- ventions most relevant to the children and youth you presently work with. Discuss what you have garnered with peers.
2. For researchers • Focus on application to real-world practice in your dissemination efforts: for example, ‘What are the top five practice implications of your latest study and where might these most usefully be dis- seminated?’ • Seek practice-partners for agency-centred research projects specifically focused on issues of implementation of evidence-based practices.
3. For child and family service agencies • Discover first what is working within the agency and build on that as a foundation before purchas- ing ‘off-the-shelf ’ models. • Develop an internal capacity to systematically analyse routinely gathered data at the case level and aggregate level and ‘mine’ this information to inform practice.
4. For the service system/policy level • Here, and speaking from a parochial perspective, with all of our resources in the USA, we are sorely in need of a new structure or body within a state or authority that ‘connects-the-dots’ between relevant service policy, research and practice in support of enhancing the implementation of evidence-based practices to improve outcomes. While the title and organizational form for such a body proves elusive – clearinghouse? executive steering committee for evidence-based programme improvement? – its key function should be to focus laser-like attention on the question of what is most important in evidence-based practice implementation in a state, local authority or region: What do we need to learn over the next 12 months? How will we learn it? How will we decide ‘what-trumps-what’: Cost? Urgent service need? Level of evidence? Cultural relevancy? Organizational compatibility? Special opportunity to experiment with proven efficacious model programmes? While the fruits of such a new body would be experienced locally, one hopes that its field of vision and, eventually, its impact would extend cross-nationally.
In sum, evidence-based practice has added greatly to the ‘tool kit’ of social services in the identification of proven efficacious models of intensive intervention such as those referenced earlier in this paper.That said, the task of scaling-up these exemplars is proving to be
Evidence-based intervention for high-risk youth J K Whittaker174 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
complex and challenging, and will require both focused attention on the multiple contextual elements that impede and enhance the adoption of evidence- based alternatives, as well as a critical re-examination of existing biases – for example, the current and often reflexive negative attitudes towards residential provi- sion in any form – that underlay current services plan- ning. Both of these conversations will be greatly enhanced by multilevel, sustained and data-oriented cross-national collaboration among practitioners, service planners and researchers. Fortunately, through technological innovations such as electronic commu- nication, the means for such collaboration are close at hand. High-resource-using youth and their families presently in, or at risk of entering the intensive services system, will be the ultimate beneficiaries of our efforts.
R E F E R E N C E S
Adams, P. & Krauth, K. (1995) Working with families and com-
munities: the patch approach. In: Reinventing Human Services:
Community and Family Centered Practice (eds P. Adams & K.
Nelson), pp. 87–109. Transaction-Aldine Publishers, Piscat-
away, NJ.
Barbarin, O., McCandies, T., Coleman, C. & Atkinson, T. (2004)
Ethnicity and culture. In: Intervention with Children and Fami-
lies: An Interdisciplinary Perspective (eds P. Allen-Meares &
M.W. Fraser), pp. 27–54. Pearson,
Boston, MA.
Beauchaine, T.P., Webster-Stratton, C. & Reid, M.J. (2005)
Mediators, moderators and predictors of one-year outcomes
among children treated for early onset conduct problems: a
latent growth curve analysis. Journal of Consulting and Clinical
Psychology, 73, 371–388. Biehal, N. (2008) Preventive services for adolescents: exploring
the process of change. British Journal of Social Work, 38, 444– 461.
Blasé, K.A. & Fixsen, D.L. (2003) Evidence-based programs and
cultural competence (20pp). Available from Tampa, FL: Univer-
sity of South Florida, Louis de la Parte Florida Mental Health
Institute, The National Implementation Research Network.
Bullock, R., Courtney, M., Parker, R., Sinclair, I. & Thoburn, J.
(2006) Can the Corporate State Parent. Children and Youth
Services Review, 28, 1344–1358. Burchard, J.D., Bruns, E.J. & Burchard, M.I. (2002) The wrap-
around approach. In: Community Treatment for Youth: Evidence-
based Interventions for Severe Emotional and Behavioral Disorders
(eds B.J. Burns & K. Hoagwood), pp. 69–91. Oxford Univer-
sity Press,
New York.
Burns, B.J. & Goldman, S.K. (1999) Promising practices in wrap-
around for children with serious emotional disturbance and their
families. Systems of Care: Promising Practices in Children’s
Mental Health, 1998 series, Volume IV. Washington, DC:
Center for Effective Collaboration and Practice, American
Institutes for Research.
Burns, B.J. & Hoagwood, K. (eds) (2002) Community Treatment
for Youth: Evidence-based Interventions for Severe Emotional and
Behavioral Disorders. Oxford University
Press, New York.
Chamberlain, P. (2002) Treatment foster care. In: Community
Treatment forYouth: Evidence-based Interventions for Severe Emo-
tional and Behavioral Disorders (eds K. Hoagwood & B.J.
Burns), pp. 117–139. Oxford University Press, New York.
Chamberlain, P. (2003) Treating Juvenile Offenders: Advances
Made through the Oregon Multidimensional Treatment Foster Care
model. American Psychological Association, Washington, DC.
Chamberlain, P. & Reid, J.B. (1998) Comparison of two com-
munity alternatives to incarceration for chronic juvenile
offenders. Journal of Consulting and Clinical Psychology, 66, 624–633.
Chorpita, B.F., Becker, K. & Daleiden, E.L. (2007) Understand-
ing the common elements of evidence-based practice: miscon-
ceptions and clinical examples. Journal of the American
Academy of Child and Adolescent Psychiatry, 46, 647–652. Courtney, M.E. & Ivaniec, D. (eds) (2009) Residential Care of
Children: Comparative Perspectives. Oxford University Press,
New York and Oxford.
Egelund, T. & Lausten, M. (2009) Prevalence of mental health
problems among children placed in out-of-home care in
Denmark. Child & Family Social Work, 14, 156–165. Fixsen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M. &
Wallace, F. (2005) Implementation Research: A Synthesis of the
Literature. University of South Florida, Louis de la Parte
Florida Mental Health Institute, Tampa, FL. The National
Implementation Research Network (FMHI Publication
#231). For more information, Available at: http://www.
fpg.unc.edu/~NIRN/ (accessed 18 February 2009).
Ghate, D. & Hazel, N. (2002) Parenting in Poor Environments:
Stress, Support & Coping. Jessica Kingsley Pub
lishers,
London.
Ghate, D., Hauari, H., Hollingworth, K. & Lindfield, S. (2008)
Key Elements of Effective Practice in Parenting Support in theYouth
Justice Context: Source Document. Youth Justice Board, London.
Glisson, C., Landsverk, J., Schoenwald, S.K., Kelleher, K.,
Hoagwood, K.E., Mayberg, S. et al. (2008) Assessing the
organizational social context (OSC) of mental health services
for implementation research and practice. Administration and
Policy in Mental Health and Mental Health Services Research,
35, 98–113. Henggeler, S.W. & Lee, T. (2003) Multisystemic treatment of
serious clinical problems. In: Evidence-based Psychotherapies for
Children and Adolescents (eds A.E. Kazdin & J.R. Weisz), pp.
301–322. The Guilford Press, New York.
Henggeler, S.W., Schoenwald, S.K., Borduin, C.M., Rowland,
M.D. & Cunningham, P.B. (1998) Multisystemic Treatment of
Antisocial Behavior in Children and Adolescents. Guilford Press,
New York.Hoagwood, K., Burns, B.J. & Weisz, J.R. (2002) A profitable
conjunction: from science to service in children’s mental
health. In: Community Treatment for Youth: Evidence-based
Interventions for Severe Emotional and Behavioral Disorders (eds
B.J. Burns & K. Hoagwood), pp. 327–339. Oxford University
Press, New York. Evidence-based intervention for high-risk youth J K Whittaker175 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
http://wwwJackson, A., Frederico, M., Tanti, C. & Black, C. (2009) Explor-
ing outcomes in a therapeutic service response to the emo-
tional and mental health needs of children who have
experienced abuse and neglect in Victoria, Australia. Child &
Family Social Work, 14, 198–212. Jensen, P.J., Weersing, R., Hoagwood, K.E. & Goldman, E.
(2005) What is the Evidence for Evidence-Based Treatments?
A hard look at our soft underbelly. Mental Health Services
Research, 7, 53–74. Kazdin, A.E. (2000) Psychotherapy for Children and Adolescents:
Directions for Research and Practice. Oxford University Press,
Oxford and New York.
Kazdin, A.E. & Weisz, J.R. (2003) Evidence-based psychotherapies
for children and adolescents. The Guilford Press, New York.
Kemp, S.P., Whittaker, J.K. & Tracy, E.M. (1997) Person-
environment Practice:The Social Ecology of Interpersonal Practice.
Transaction-Aldine Publishers,
Piscataway, NJ.
Kemp, S.P., Allan-Eckard, K., Ackroyd, A., Becker, M.F. &
Burke, T.K. (2005) Community family support meetings: con-
necting families, public child welfare and community
resources. In: Child Welfare for the 21’st Century: A Handbook of
Practice (eds G.P. Mallon & P.M. Hess), pp. 102–117. Colum-
bia University Press, New York.
Lau, A.S. (2006) Making the case for selected and directed
cultural adaptations of evidence-based treatments: examples
from parent training. Clinical Psychology: Science and Practice,
13, 295–310. Lightburn, A. & Sessions, P. (eds) (2006) Handbook of
Community-Based Clinical Practice. Oxford University Press,
Oxford and New York.Littell, J.H. (2005) Lessons from a systematic review of effects of
multisystemic therapy. Children and Youth Services Review, 27, 445–463.
Littell, J.H. (2008) Evidence-based or biased? The quality of
published reviews of evidence-based practices. Children and
Youth Services Review, 30, 1299–1317. McAuley, C. & Davis, T. (2009) Emotional well-being and
mental health of looked after children in the UK. Child &
Family Social Work, 14, 147–155. McAuley, C., Pecora, P.J. & Rose, W. (eds) (2006) Enhancing the
Well-being of Children and Families through Effective Interven-
tions: International Evidence for Practice. Jessica Kingsley Pub-
lishers, London.Macdonald, G. (2001) Effective Interventions for Child Abuse and
Neglect. John Wiley & Sons Ltd., New York.
Miranda, J., Guillermo, B., Lau, A., Kohn, L., Wei-Chin, H. &
LaFromboise, T. (2005) State of the science on psychosocial
interventions for ethnic minorities. Annual Review of Clinical
Psychology, 1, 113–142. Munro, E., Stein, M. & Ward, H. (2005) Comparing how dif-
ferent social, political and legal frameworks support or inhibit
transitions from public care to independence in Europe, Israel,
Canada and the United States. International Journal of Child
and Family Welfare, 8, 197–198. Pecora, P.J., White, C.R., Jackson, L.J. & Wiggins, T. (2009a)
Mental health of current and former recipients of foster care:
a review of recent studies in the USA. Child & Family Social
Work, 14, 132–146. Pecora, P.J., Whittaker, J.K., Maluccio, A.N., Barth, R.P. &
DePanfilis, D. (2009b) The ChildWelfare Challenge: Policy, Prac-
tice and Research, 3rd edn. Transaction-Aldine Publishers,
Piscataway, NJ.Pennell, J. & Anderson, G.R. (2005) Widening the Circle: The
Practice and Evaluation of Family Group Conferencing with Chil-
dren,Youths and Their Families. NASW Press, Washington, DC.
Price, J.M., Chamberlain, P., Landsverk, J. & Reid, J. (2009)
KEEP foster parent training intervention: model description
and effectiveness. Child & Family Social Work, 14, 233–242. Rauktis, M.E., De Andrade, V., Doucette, A.R., McDonough,
A. L. & Reinhart, S. (2005) Treatment foster care and rela-
tionships: understanding the role of therapeutic alliance
between youth and treatment parent. International Journal of
Child & Family Welfare, 8, 146–163. Sameroff, A.J. & Gutman, L.M. (2004) Contributions of risk
research to the design of successful interventions. In: Interven-
tion with Children and Families: An Interdisciplinary Perspective
(eds P. Allen-Meares & M.W. Fraser), pp. 9–27. Pearson,
Boston, MA.Schoenwald, S.K. & Rowland, M.D.v2002) Multisystemic
therapy. In: Community Treatment forYouth: Evidence-based Inter-
ventions for Severe Emotional and Behavioral Disorders (eds B.J.
Burns & K. Hoagwood), pp. 91–116. Oxford University Press,
New York.Schorr, L.B. (1993) Effective Strategies for Increasing Social
Program Replication/Adaptation. Seminar on Effective Strategies
For Increasing Social Program Replication/Adaptation. National
Association of Social Workers, Washington, DC.
Sinclair, I. (2006) Residential care in the UK. In: Enhancing the
Well-being of Children and Families through Effective Interven-tions: International Evidence for Practice (eds C. McAuley, P.J.
Pecora & W. Rose), pp. 203–216. Jessica Kingsley Publishers,
London.Stroul, B.A. & Friedman, R.M. (1986) A System of Care for
Severely Emotionally Disturbed Children & Youth. CASSP Tech-
nical Assistance Center, Georgetown University, Child Devel-
opment Center, Washington, DC.
Thoburn, J. (2007) Globalization and Child Welfare: Some Lessons
from Cross-national Study of Children in Out-of-Home Care.
Social Work Monographs, University of East Anglia, Norwich,
England.
US Department of Health and Human Services (1999) Mental
Health: A Report of the Surgeon GENERAL. US Department of
Health and Human Services, Substance Abuse and Mental
Health Services Administration, National Institutes of Health,
National Institute of Mental Health, Rockville, MD.
Weisz, J.R. (2004) Psychotherapy for Children and Adolescents:
Evidence-based Treatments and Case Examples. Cambridge
University Press, Cambridge.
Weisz, J.R. & Gray, S. (2008) Evidence-based psychotherapy
for children and adolescents: data from the present and a
model for the future. Child and Adolescent Mental Health, 13, 54–65.
Evidence-based intervention for high-risk youth J K Whittaker176 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
Whittaker, J.K. (2005) Creating ‘prosthetic environments’ for
vulnerable children: emergent cross-national challenges for
traditional child & family services practice. In: In the Best
Interests of Children and Youth: International Perspectives (eds H.
Grietens, W. Hellinckx & L. Vandemeulebroecke), pp. 99–119.
Leuven University Press, Leuven.
Whittaker, J.K. (2006) Residential care in the US. In: Enhancing
the Well-being of Children and Families through Effective Interven-
tions: International Evidence for Practice (eds C. McAuley, P.J.Pecora & W. Rose), pp. 217–227. Jessica Kingsley Publishers,
London.Whittaker, J.K. & Maluccio, A.N. (2002) Re-thinking ‘child
placement’: a reflective essay. Social Service Review, 76, 108– 134.
Whittaker, J.K., Greene, K., Schubert, D., Blum, R., Cheng, K.,
Blum, K. et al. (2006) Integrating evidence-based practice in
the voluntary child mental health agency: a template for clini-
cal and organizational change. American Journal of Orthopsy-
chiatry, 76, 194–201.
Evidence-based intervention for high-risk youth J K Whittaker177 Child and Family Social Work 2009, 14, pp 166–177 © 2009 Blackwell Publishing Ltd
elines for Selecting an Evidence‐Based Program
What Works, Wisconsin – Research to Practice Series, #3
In recent years there has been a significant increase in the number of evidence‐
based programs designed to reduce individual and family problems and
promote healthy development. Because each program has undergone rigorous
testing and evaluation, program practitioners can reassure potential program
sponsors that the program is likely to be effective under the right conditions, with
the appropriate audience and with the proper implementation. However, knowing
which program is the “right” one for a particular setting and audience is not always
easy to determine. When selecting a program, it is important to move beyond current
fads or what the latest salesperson is selling and consider whether a program fits with
the local agency’s goals and values, the community setting and the needs of the
targeted audience. The long‐term success of a program depends on the program being
not only a good one, but also the right one.
Unfortunately, there is currently little research on how to best go about the process of
selecting an evidence‐based program. Consequently, the guidelines we present in this brief
are based primarily on our experiences working with community‐based organizations, the
experiences of practitioners, and common sense. We have identified a number of factors that
we believe should be considered when deciding which program is the most appropriate for a
particular audience and sponsoring organization. These factors can be grouped into three
general categories: program match, program quality and organizational resources. In order to
assist with the process of program selection, we have developed a set of questions to consider
when selecting an evidence‐based program for your particular agency and audience.
WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES
Guidelines for selecting an evidence‐based program:
Balancing community needs, program quality,
and organizational resources
ISSUE #3, MARCH 2007
BY STEPHEN A. SMALL, SIOBHAN M. COONEY,
GAY EASTMAN, AND CAILIN O’CONNOR
University of Wisconsin–Madison and University of Wisconsin–Extension
Guidelines for Selecting an Evidence‐Based Program 2
What Works, Wisconsin – Research to Practice Series, #3Program match:
Questions to ask
How well do the program’s goals and objectives reflect what your organization hopes to achieve?
How well do the program’s goals match those of your intended participants?
Is the program of sufficient length and intensity (i.e., “strong enough”) to be effective with this particular
group of participants?
Are potential participants willing and able to make the time commitment required by the
program?
Has the program demonstrated effectiveness with a target population similar to yours?
To what extent might you need to adapt this program to fit the needs of your community? How
might such adaptations affect the effectiveness of the
program? Does the program allow for adaptation?
How well does the program complement current programming both in your organization and in the
community?
The issues raised by program match, program
quality and organizational resources are overlap‐
ping. Selecting a program usually requires
balancing different priorities, so it’s important to
have a good understanding of all three of these
before determining the usefulness of a program
for a particular situation.
PROGRAM MATCH A first set of factors to consider is related to how
well the program will fit with your purposes, your
organization, the target audience, and the com‐
munity where it will be implemented.
Perhaps the most obvious factor to consider is
whether the goals and objectives of a program are
consistent with the goals and objectives that the
sponsoring organization hopes to achieve. While
this may seem apparent, it is not uncommon for
sponsors to select a program because there is grant
money available to support it or everyone else is
doing it. Just because a program is the latest fad or
there’s funding to support it doesn’t necessarily
mean it is going to accomplish the goals of the
sponsoring organization or meet the needs of the
targeted audience.
A second aspect of program match involves
whether a program is strong enough to address
the level and complexity of risk factors or current
problems among participants. This refers to the
issue of adequate program duration and intensity.
Changing existing problem behaviors or counter‐
acting a large number of risk factors in partici‐
pants’ lives requires many hours of engaging
programming over a period of time. For example,
a short primary prevention program designed for
families facing few problems or risks may not be
effective for an audience already experiencing
more severe problems.
Another facet of program match concerns the
length of the program and whether your intended
audience will be willing and able to attend the
required number of sessions. Many evidence‐
based programs are of fairly long duration,
involving multiple sessions over weeks or
months. A common concern of program pro‐
viders is whether potential participants will
make such a long‐term commitment. Because
this is a realistic concern, program sponsors need
to assess the targeted audience’s availability for
and interest in a program of a particular length.1
The reality is, if people don’t attend, then they
can’t reap the program’s benefits. However, it is
also important to keep in mind that programs of
longer duration are more likely to produce
lasting behavior change in participants. Program
sponsors sometimes need to find a compromise
between the most effective program and one that
will be a realistic commitment for
participants.
Matching a program with the values and culture
of the intended audience is also critically import- ant. Some programs are intentionally designed
for particular populations or cultural groups.
Most are more culturally generic and designed
1 Issue #2 in this series addresses strategies for recruiting and retaining participants.
Guidelines for Selecting an Evidence‐Based Program 3
What Works, Wisconsin – Research to Practice Series, #3Program quality: Questions to ask
Has this program been shown to be effective? What is the quality of this evidence?
Is the level of evidence sufficient for your organization?
Is the program listed on any respected evidence‐ based program registries? What rating has it
received on those registries?
For what audiences has the program been found to work?
Is there information available about what adaptations are acceptable if you do not
implement this program exactly as designed? Is
adaptation assistance available from the program
developer?
What is the extent and quality of training offered by the program developers?
Do the program’s designers offer technical assistance? Is there a charge for this assistance?
What is the opinion and experience of others who have used the program?
for general audiences.2 It’s important to consider
whether the targeted audience will find the
program acceptable and will want to participate.
The ideal situation would be finding evidence that
a program is effective for the specific pop- ulation(s) you intend to use it with. In that case, you could reasonably expect the program to be
effective when it is implemented well.
Unfortunately, many evidence‐based programs
have only been evaluated with a limited number
of populations and under a relatively narrow
range of conditions. While many evidence‐based
programs are effective and appropriate for a range
of audiences and situations, it is rare to find a
program that is suitable or effective for every
audience or situation. In many cases, you will need to carefully read program materials or talk to
the program’s designers to see whether adapting a
program or using it with an audience for which it
hasn’t been evaluated is reasonable.
Depending on the design, programs may or may
not be amenable to adaptation. If adapting a
program to a particular cultural group is
important, then program sponsors should serious- ly consider whether such changes are possible.
Some program designers are willing to help you
with program adaptation so that the program’s
effectiveness will not be undermined by these
changes.3
Finally, when considering which program to
select, sponsors should consider whether the pro- gram complements other programs being offered
by the sponsoring organization and by other
organizations in the community. The most
effective approaches to prevention and inter- vention involve addressing multiple risk and
2 Issue #1 in this series addresses the issue of culture and evidence‐based programs. 3 Issue #4 in this series will address issues of program fidelity and adaptation.
protective factors, developmental processes and
settings. Any new program implemented in a
community should address needs that other
community programs fail to address, which will
help to create the kind of multi‐pronged approach
that leads to greater overall effectiveness.
PROGRAM QUALITY A second set of factors to consider when selecting
a program are related to the quality of the pro‐
gram itself and the evidence for its effectiveness.
The program should have solid, research‐based
evidence showing that it is effective. For a pro‐
gram to be deemed evidence‐based, it must go
through a series of rigorous evaluations. Such
evaluations have experimental or quasi‐experi‐
mental designs – meaning they compare a group
of program participants to a similar group of
people who did not participate in the program to
determine whether program participation is assoc‐
iated with positive changes. These kinds of eval‐
Guidelines for Selecting an Evidence‐Based Program 4
What Works, Wisconsin – Research to Practice Series, #3TABLE 1: Selected evidence‐based program registries
Blueprints for Violence Prevention
http://www.colorado.edu/cspv/blueprints/index.html
This registry is one of the most stringent in terms of endorsing programs as Model or Promising. Programs are
reviewed by an expert panel and staff at the University of Colorado, and endorsements are updated regularly.
Programs are added and excluded from the registry based on new evaluation findings.
Helping America’s Youth
http://guide.helpingamericasyouth.gov/programtool.cfm
This registry was developed with the help of several federal agencies. Programs focus on a range of youth
outcomes and are categorized as Level 1, Level 2, or Level 3 according to their demonstrated effectiveness. The
registry is updated regularly to incorporate new evidence‐based programs.
Office of Juvenile Justice and Delinquency Prevention Model Program Guide
http://www.dsgonline.com/mpg2.5/mpg_index.htm
This registry is one of the largest currently available and is continuously updated to include new programs.
Programs found on this registry are designated as Exemplary, Effective, or Promising.
Promising Practices Network
http://www.promisingpractices.net/
A project of the RAND Corporation, this registry regularly updates its listings of Effective and Promising
programs. Programs are reviewed and endorsed by project staff.
Strengthening Americaʹs Families
http://www.strengtheningfamilies.org/html/
Although this registry was last revised in 1999, it is the only registry with a focus specifically on family‐based
programs. Programs were reviewed by expert panels and staff at the University of Utah and the Center for
Substance Abuse Prevention. They were then designated as Exemplary I, Exemplary II, Model, or Promising.
Substance Abuse and Mental Health Services Administration (SAMHSA) National Registry of Evidence‐
Based Programs and Practices
http://www.nrepp.samhsa.gov
This recently re‐launched site no longer categorizes programs as Model, Effective, or Promising. Instead,
programs are summarized and the quality of the research findings is rated separately for each outcome that has
been evaluated. SAMHSA has also introduced a “Readiness for Dissemination” rating for each reviewed program.
Nominations are accepted each year for programs to be reviewed; SAMHSA funds independent consultants to
review nominated programs and update the registry.
uations allow for a reasonable assumption that it
was the program itself that changed people’s
knowledge, attitudes or behavior.
As funders and program sponsors become more
committed to implementing evidence‐based pro‐
grams, program developers are increasingly likely
to promote their programs as evidence‐based.
However, just because a program developer ad‐
vertises a program as evidence‐based doesn’t
mean that it meets the standards discussed above.
For example, a program might be “research‐
based,” but not “evidence‐based.” A research‐
based program has been developed based on
research about the outcomes or processes it add‐
resses. However, it has probably not been
subjected to the rigorous evaluations and real‐
world testing that are needed to designate a
program as evidence‐based. The simplest way to
determine evidence of a program’s effectiveness is
Guidelines for Selecting an Evidence‐Based Program 5
What Works, Wisconsin – Research to Practice Series, #3Organizational resources:
Questions to ask What are the training, curriculum, and implementation costs of the program?
Can your organization afford to implement this program now and in the long‐term?
Do you have staff capable of implementing this program? Do they have the qualifications
recommended or required to facilitate the
program? Would your staff be enthusiastic about a program of this kind and are they willing to make the
necessary time commitment?
Can this program be implemented in the time available?
What’s the likelihood that this program will be sustained in the future?
Are your community partners supportive of your implementation of this program?
to examine the designations given by well‐estab‐
lished and respected evidence‐based program
registries. Program registries classify programs at
different levels of endorsement based on evidence
of effectiveness for certain participant outcomes.
See Table 1 for an annotated listing of program
registries.
If a program is not listed on a respected registry,
then it is important to seek out scientific evidence
of the program’s effectiveness. At a minimum, you
should review any evaluation studies that have
been conducted by the program developer and
external evaluators. Ideally, these evaluations use
an experimental or quasi‐experimental research
design. Another sign of a high‐quality evaluation
is that its results have been published in a well‐
respected, peer‐reviewed, scientific journal.
An additional indicator of program quality to
consider is the level of training and follow‐up
support available from the program designers.
Some programs have a great deal of resources
available to help program implementers. These
resources can be especially important if you’re
working with a unique audience and need to
make adaptations or if program implementation is
particularly complex. As a general rule, more in‐
tensive training and more follow‐up support from
the program developer will increase the effective‐
ness and sustainability of a program over time.
Some programs provide excellent technical assis‐
tance; staff members are accessible and willing to
address questions that arise while the program is
being implemented. Often this technical assistance
is free, but sometimes program designers charge
an additional fee for it. Therefore, the benefits and
costs of technical assistance should be kept in
mind when selecting an evidence‐based program.
Finally, while the scientific literature and infor‐
mation from the program developer provide key
information about program quality, don’t over‐
look the experience of practitioners who have imp‐
lemented the program. Ask whether they
encountered any obstacles when implementing
the program, whether they believe the program
was effective, which audiences seemed to respond
most positively to the program, and whether they
would recommend the program for your sit‐
uation. This type of information is usually not
included in scientific program evaluations but is a
critically important consideration for most
practitioners.
ORGANIZATIONAL
RESOURCES A final set of factors to consider when selecting a
program is related to the resources required for
carrying out the program. Consider whether your
organization has the expertise, staff, financial sup‐
port and time available to implement the pro‐
gram. Implementing evidence‐based programs is
usually fairly time‐ and resource‐intensive. For
example, evidence‐based programs often require
facilitators to attend multi‐day trainings or call for
facilitators with particular qualifications. Even if a
program is a good fit for your community, if your
organization doesn’t have the human or financial
resources to adequately implement the program,
its chances of success are limited.
Guidelines for Selecting an Evidence‐Based Program 6
What Works, Wisconsin – Research to Practice Series, #3WHAT WORKS, WISCONSIN: RESEARCH TO PRACTICE SERIES
This is one of a series of Research to Practice briefs prepared by the What Works, Wisconsin team at the
University of Wisconsin–Madison, School of Human Ecology, and Cooperative Extension, University of
Wisconsin–Extension. All of the briefs can be downloaded from: http://whatworks.uwex.edu
This series expands upon ideas that are discussed in What Works, Wisconsin: What Science Tells Us about Cost‐
Effective Programs for Juvenile Delinquency Prevention, which is also available for download at the address above.
This publication may be cited without permission provided the source is identified as: Small, S.A., Cooney,
S.M., Eastman, G., & O’Connor, C. (2007). Guidelines for selecting an evidence‐based program: Balancing
community needs, program quality, and organizational resources. What Works, Wisconsin Research to Practice
Series, 3. Madison, WI: University of Wisconsin–Madison/Extension.
This project was supported by Grant Award No. JF‐04‐PO‐0025 awarded by the Wisconsin Office of Justice
Assistance through the Wisconsin Governor’s Juvenile Justice Commission with funds from the Office of
Juvenile Justice and Delinquency Prevention.
The authors wish to thank Mary Huser of the University of Wisconsin–Extension for her edits, comments, and
suggestions in the development of this Research to Practice brief.
In addition, when selecting a program it makes
sense to assess your organization’s long‐term
goals and consider which programs have the best
chance of being continued in the future. Programs
that require significant external funding are
especially prone to abandonment after the funding
runs out. Some programs are more readily adopt‐
ed by existing organizations and are easier to
support over the long run. Think about whether a
program has a good chance of being integrated
into the base programming of your organization.
Can the program be continued in the future with
existing staff and resources or will it always
require external support?
Lastly, because many evidence‐based programs
are resource intensive, think about collaborating
with other organizations in the community to
deliver a program. Selecting a program that meets
the needs of two or more agencies may allow for
the pooling of resources, thus enhancing the
likelihood that the program can be adequately
funded, implemented and sustained over time.
Additionally, such an arrangement can lead to
positive, long‐term partnerships with other com‐
munity agencies.
While all three of these factors are important,
some may be more crucial to your organization
than others. The key to selecting the best program
for your particular situation involves balancing
different priorities and trade‐offs and finding a
program that best meets these competing
demands. By selecting a high quality program that
matches the needs of your audience and com‐
munity and the resources of your organization,
you greatly enhance the likelihood that you will
have an effective program that will have a long‐
term impact and improve the lives of its
participants.Order an Essay Now & Get These Features For Free:
Turnitin Report
Formatting
Title Page
Citation
Outline
