Term
|
Definition
Increase your knowledge of processes involved in program evaluation
Provide information and resources to help you design and conduct your own program evaluation |
|
|
Term
|
Definition
Why evaluate?
What is evaluation?
What does evaluation do?
Kinds of evaluation |
|
|
Term
|
Definition
Determine program outcomes
Identify program strengths
Identify and improve weaknesses
Justify use of resources
Increased emphasis on accountability
Professional responsibility to show effectiveness of program |
|
|
Term
|
Definition
| Purposeful, systematic, and careful collection and analysis of information used for the purpose of documenting the effectiveness and impact of programs, establishing accountability, and identifying areas needing change and improvement |
|
|
Term
|
Definition
| Looks at the results of your investment of time, expertise, and energy, and compares those results with what you said you wanted to achieve |
|
|
Term
|
Definition
Process
Formative
Impact
Summative
Outcome |
|
|
Term
|
Definition
| Documents what the program is and to what extent it has been implemented |
|
|
Term
|
Definition
| A new program is being introduced; identifies and defines the program; identifies what you are actually evaluating |
|
|
Term
| Process evaluation: example |
|
Definition
| Who receives program, where is program operating; is it being implemented the same way at each site? |
|
|
Term
|
Definition
| What short-term or immediate effect did the program have? |
|
|
Term
|
Definition
| Collect data for evaluation at the beginning and end of program to ID changes (pre-post data collection) |
|
|
Term
| Impact evaluation: example |
|
Definition
| Collecting pre-post data on knowledge change, changes in health service utilization, policy changes, or other data |
|
|
Term
|
Definition
| Identifies the results or effects of a program |
|
|
Term
|
Definition
| You want to measure students’ or clients’ knowledge, attitudes, and behaviors as a result of a program |
|
|
Term
| Outcome evaluation: example |
|
Definition
| Did program increase achievement, reduce truancy, create better decision-making? |
|
|
Term
|
Definition
Conducted during Process Evaluation
As the program is happening to make changes as program is being implemented |
|
|
Term
|
Definition
Conducted during Impact Evaluation
At the end of a program to document results |
|
|
Term
| The evaluative process (4) |
|
Definition
Planning
Development
Implementation
Feedback |
|
|
Term
|
Definition
|
|
Term
| Step 1: Define Purpose & Scope |
|
Definition
Why are you doing the evaluation?
Mandatory? Program outcomes? Program improvement?
What is the scope? How large will the effort be?
Large/Small; Broad/Narrow
How complex is the proposed evaluation?
Many variables, many questions?
What can you realistically accomplish? |
|
|
Term
|
Definition
Resources
$$
Staff
Who can assist?
Need to bring in expertise?
Do it yourself?
Advisory team?
Time
Set priorities
How you will use the information |
|
|
Term
|
Definition
What is it that you want to know about your program?
Operationalize it (make it measurable)
Do not move forward if you cannot answer this question. |
|
|
Term
|
Definition
Strategic plans
Mission statements
Policies
Needs assessment
Goals and objectives
National standards and guidelines |
|
|
Term
| Identifying Evidence Needed to Answer Questions |
|
Definition
What evidence do you have to answer your question?
Need to think about what information you need in order to answer your evaluation questions |
|
|
Term
| Specify Evaluation Design |
|
Definition
Relates to when data should be collected
Types of Designs
Decide what type of design fits each question |
|
|
Term
|
Definition
Status (here and now; snapshot)
Comparison (group A vs. group B; program A vs. program B)
Change (what happened as a result of a program; what differences are there between time A and time B)
Longitudinal (what happens over extended time) |
|
|
Term
| Components of Data Collection Action Plan |
|
Definition
What Will be Collected?
Based on evidence required
How Collected? Instrumentation
Surveys? Published instrument? Focus group? Observations?
From Whom Collected?
Who or what provides evidence
When Collected and by Whom?
Specific dates, times, persons
How Data are to be Analyzed? |
|
|
Term
|
Definition
Students
Parents
Teachers
Counselors
Employers
Friends
Documents and other records |
|
|
Term
|
Definition
Commercial instrument
Survey/questionnaire
Focus group/interviews
Observations
Archived information |
|
|
Term
|
Definition
Sometimes best to use published or research instruments
Why???
Particularly for tough constructs
Since its not made specifically for you, may not answer your question entirely |
|
|
Term
| Self-Constructed Instruments Questionnaires |
|
Definition
Focus on evidence you need
Use simple language
Ask only what you need; keep it short
Don’t use jargon
Each question should focus on one idea
Make sure terms are clear
Make it easy for person to answer the questions (check rather than write, where possible)
Use extended response when you want details |
|
|
Term
| Self-Constructed Instruments Focus Groups / Interviews |
|
Definition
Good to use when you want extended and detailed responses
Craft an agenda and stick to it
Keep groups small (6-10); time short (1-1.5 hours)
Specify objectives of session
Questions need to be clear; one question at a time
Encourage everyone to participate
Use opportunity to probe deeper on a topic |
|
|
Term
| Data: Archives & Documents |
|
Definition
| examine whats already available |
|
|
Term
|
Definition
How much data should you collect?
100% of target audience is ideal; may be too expensive and time consuming
If not 100%, sample is OK if group is representative of group as a whole (population) |
|
|
Term
|
Definition
Data collected during program evaluation are compiled and analyzed (counting; number crunching)
Inferences are drawn as to why some results occurred and others did not
Can be very complex depending on your evaluation questions
We will focus on simple things that can be done without expert consultants |
|
|
Term
|
Definition
Descriptive data analyses
Used to describe, classify, and summarize data
E.g. Mean, median, mode
What was the average score on the cholesterol knowledge test? |
|
|
Term
|
Definition
Measures of relationship or correlation
Establish a relationship between 2 variables
E.g. ANOVA, Chi Square
Do students’ attitudes about bicycle helmets differ in rural and urban settings? |
|
|
Term
| Multivariate Data Analysis |
|
Definition
Multiple analyses are used to determine relationships between more than 2 variables
Multiple Regression
Can the risk of heart disease be predicted using smoking, exercise, diet, and heredity? |
|
|
Term
|
Definition
Examine results carefully and objectively
Draw conclusions based on your data
What do the results signify about your program? |
|
|
Term
| What to include in documentation |
|
Definition
Program description
Evaluation questions
Methodology (how and from whom and when)
Response rate
Methods of analysis
Conclusions listed by evaluation question
General conclusions and findings
Action items
Recommendations for program improvement and change |
|
|
Term
| Document Successes and Shortfalls |
|
Definition
Highlight and brag about positive outcomes
Document shortfalls
Provides opportunities to
Improve program
Make recommendations to benefit the program |
|
|
Term
|
Definition
Determine disseminating methods
Inform all your relevant stakeholders on results
Dissemination methods should differ by your target audience |
|
|
Term
|
Definition
Your program staff
Businesses
Partners that work with your program
Employers
School Level
School administrators
Counselors
Teachers
Students
Parents
Media
Local newspaper
TV station
Radio program
Community or school newsletter
Education Researchers
Members of Community or faith based organizations
Church members
Religious leaders
Rotary club
Boys or girls club
Anyone who participated in your evaluation! |
|
|
Term
|
Definition
Reports
Journal articles
Conferences
CRN newsletter/tabloids
Presentations
Brochures
TV and newspaper interviews
Executive summary
Posting on Web site |
|
|
Term
| Feedback for Program Improvement |
|
Definition
Evaluation results may show areas where improvement is necessary
Opportunity to fix shortfalls
- 25% of 11th graders are unable to complete a skills based resume
- 85% of our students drop out of college in the first year
- Most employers do not want your students to serve an interns in their companies |
|
|
Term
| You can use evaluation findings to make program improvements |
|
Definition
Consider adjustments
Re-examine/revise program strategies
Change programs or methodologies
Increase time with the program
Use your results as a needs assessment for future efforts |
|
|
Term
|
Definition
Determine the effects of the program on recipients
Know if you have reached your objectives
Improve your program |
|
|
