Outcomes are the changes you expect to result from your program. These can be changes in individuals, families, systems, policies, or institutions that you are trying to alter. Outcomes are often confused with outputs – outputs are the direct products of the activities undertaken to reach the desired outcomes. In the domestic violence field, outputs are often programs, number of trainings that occurred, materials distributed, and number of families served; whereas outcomes are things like knowledge transferred, behaviors changed, symptoms reduced, etc.
When considering how best to measure the success of a program or project, outcomes should be directly related to the goals and objectives. Program goals and objectives establish criteria against which you can determine program impact and effectiveness. It’s important to first identify the goals and objectives of the program or intervention you plan to evaluate. Logic models are a useful tool that can help you do this. Project-focused goals and objectives should be written as SMART goals to help you meet the long-term goals of your program. According to SAMSHA, “setting SMART objectives keeps the project moving forward, helps with accountability and timing, and lets you know that you are accomplishing what you set out to accomplish.” Goals should be specific, measurable, achievable, relevant, and time-bound.
Image retrieved from Adobe Stock Images
What do we want to know?
Why do we want to know it?
What will we do with the information?
Why are we trying to achieve this goal?
Who will do the work?
Are the words clear?
Can the goal be quantified?
How can you measure progress?
How will you know when you have achieved your goal?
How will you track progress?
What methods will be used?
What resources (e.g., funds, people, resources) do you need to meet your goal?
Is the team capable of achieving this goal?
Does the team have the skills that are needed?
Is it possible to acquire the skills and resources needed?
Relevant (also Realistic)
Why should the program achieve this goal?
Why is it important to the agency/team/community?
How does it relate to the agency’s other plans/projects and mission?
What are the impacts (on staff, on clients, in the wider community)?
By when does the team need to achieve this goal?
Are there smaller objectives that will be met along the way?
How will the team track time and achievements?
Considerations to Explore From Learning From Action to Define Outcomes:
What are the changes that you can reasonably expect the target population to experience as a result of going through your program?
How will they be different or better off as a result?
Are the outcomes logically related to and likely to result from the core program components?
Are your outcomes realistic and attainable given the level of intensity and duration of the program?
Have you moved beyond outcomes metrics and client satisfaction in developing your end-result outcomes (to include skills, behavior, knowledge gained)?
How to make a basic goal a SMART goal
Basic Goal: Provide parent and caregiver training groups in order to increase parenting competencies.
SMART Goal: In order to increase parenting competencies, provide parent and caregiver training groups in outpatient settings for 20 families in Years 1 and 2, 30 families in Years 3 and 4, and 40 families in Year 5, for a total of 140 unduplicated parents and caregivers. At least 75% of participants will increase parenting competencies according to training evaluation surveys.
There should be synergy between goals and objectives and the measures and instruments used to demonstrate that favorable outcomes have been achieved. For example, if the efficacy of a clinical intervention is being measured by the Trauma Symptom Checklist for Children (TSCC) as part of the project design, then performance indicators should be specific to what is measured by the TSCC.
Be specific about what is being measured and by the instrument being used.
Don’t say: As a result of treatment, children will feel better.
Instead say: As a result of treatment, children will show reduced symptoms of anger, anxiety, and depression as evidenced by the Trauma Symptom Checklist for Children, with at least 80% of children showing reduced symptoms after completing treatment.
SMART goals provide a framework for measurement of outcomes. The Promising Futures Guiding Principle on Storytelling encourages us to “capture stories and spread their impact using a wide range of interpersonal, cultural, and research and evaluation approaches…and to ensure different forms (qualitative, quantitative) and methods of collection (experimental, narrative, participatory action) are utilized in research and evaluation.” Ensuring that evaluation strategies are trauma informed and culturally relevant to the families you are working with is essential to gathering meaningful and accurate outcome data.
Surveys are the often the primary means of data collection in most programs, as data is often easy to collect from many people in a short amount of time. Descriptive statistics are generally enough analysis of surveys to show that outcomes have been achieved. Surveys help the team quantify opinions of participants, consumer satisfaction, utility of services, and other related topics. Other quantitative analyses are completed when using standardized and validated measures such as those used with clinical interventions. Then, data will be aggregated to show trends in symptom improvement as a result of treatment completion. Consider – written surveys must attend to literacy levels and be offered in languages that reflect the populations you are working with. Also conducting written surveys with communities that value oral communication over written may not be the most effective strategy to get accurate information about your program.
Qualitative data collection methods include individual semi-structured key informant interviews, open-ended surveys (also known as structured interviews), group interviews, focus groups, and observations. Qualitative data collection allows for more in-depth answers to questions and goes beyond what happens to why and how something happens. Most interviews will be semi-structured; that is, interviews will follow a protocol, but can be flexible. These interviews maximize data collection efficacy by making sure that pertinent questions are definitely asked but also allowing the interviewer to follow an unexpected train of thought. True focus groups require a specific format and group composition in which group members will ask each other questions to further the goals of the group facilitator. Most evaluators use simple group interviewing techniques to get the opinions of many similar individuals all at once. Most group interviews will be semi-structured, and a good facilitator will encourage all participants to speak.
Tips for Interviews
Put questions in an order that prioritizes the information you need to know and leads to a natural conversational flow
Follow-up on any answers or comments that need clarification
Probe to get more nuanced information
Avoid offering your own clarifications – just ask
Give people ample time to answer before moving on to the next question
Avoid letting people go too far off topic
While all data can be used to help programs tell their story, qualitative data is particularly useful as the information collected is so rich and provides so much more nuance and texture. (See storytelling module)
Tips for Storytelling
Include as many details as possible without giving away the identity of the participant
Include context – stories are about a specific time and place
Keep the emotion intact as much as possible
Make the descriptions vivid
Make the readers/listeners feel they are there
Ensure informed consent
Tips for Writing Results
Be thoughtful about your audience
- Who are you reporting to?
- Why are you reporting?
- What format would best reach my intended audience? Report, video, Podcast, etc.
What do you hope to accomplish by sharing results?
- Summary for community
- Internal summary for staff
- Bring to light aspects of the program that might need to change – CQI
- Demonstrate effectiveness of program
- Building evidence
- Showing funders how funding was used
- Provide rationale for shifting or adapting program strategies
- Gaining additional funding
Outcome Reporting should always include:
Explanation of the program or project
Restating of the goals and objectives
What was done to meet goals and objectives
Results of intervention and for whom
Limitations of the project or data collection and analysis methods
Thoughts about why the outcomes are significant
Recommendations for future work
- Outcome Evaluations: The Basics
- Sharing Your Impact
- SMART goals: A step-by-step guide
- SMART Goals: A How to Guide
- Assessing Community Needs and Resources; Conducting Surveys
- Assessing Community Needs and Resources: Conducting Interviews
- A Short Guide for Conducting Research Interviews
- Qualitative Research: Introducing focus groups