Assesses a process or program to privide evidence and feedback to the program.

The comparison of an object of interest against a standard of acceptability.

Is an organized process foundedjupon the scientific method for investigating problems.  It involves systematic progression through a series of necessary steps.

Referes to the consistencey, dependability, and stability of the measurement process.

An emperical estimate of the extent to which an instrument produces the same result (measure or score), applied once or two or more times.

Is the degree to which a test or assessment measures what it is intended to measure.  Using an acceptable (valid) instrument increases the chance of measuring what was intended.

Ensuring a study is technically sound.

Are operational forms of a construct.  They designate how the construct will be measured in designated scenarios.

A construct, cahracteristic, or attribute that can be measured orobserved.

1.  Was the purpose of the study stated?

2.  Was the research question or hypothesis stated?

3.  Were teh subjects in ht estudy described? Did the literature describe subject recuritment?

4.  Was the design and location of the study described?

5.  Were the data collection instruments described?

6.  Did the presented results reflect the research question or hypothesis?

7.  Were the conclusions relfective of the reserch design and data analysis?

8.  Were the implications meaningful to the priority population?

1.  determine the purpose and ofjectives of the instrument

2.  review existing instruments

3.  conduct and early review with colleagues

4.  conduct a review with a panel of experts

5.  pilot test the instrument with an appropriate sample population

6.  revise the instrument based on the steps listed above

Looks at the ongoing process of evaluation while the program is being developed and implemented. 

Any combination of measurements obtained and judgements made before or during the implementation of materials, methods, activities, or programs to control, assure or improve the quality of perfromance or delivery.

Associated with quantitative processes.  Also commonly associated with impact and outcome evaluation.

Any combination of measurement and judgements that permit comclusions to be drawn about impact, outcome or benefits of a program or method.

Considers the instrument's items of measurement for the relevant areas of interest.

The assessment of the correspondence between the items composing the insturment and the content domain from which the items were selected.

If, on the face, the measure appears to measure what it is supposed to measure.

Refers to one measure's correlation to another measure of a particular situation.

The extent to which data generated form a measurement instrument are correlated with the data generated form a measure (criterion) of the phenomenon being studied, usually an individual's behavior or performance.

Ensures that the concepts of an instrument relate to the concepts of a particular theory.

The assessment of the correspondence between the items composing the insturment and the content domain from which the items were selected.

1.  Develop instrument specidications

2.  Develop instructions for implementing the instrument and examples of how to complete items

3.  establish item scoring procedures

4.  Conduct an item analysis and reliability and validity tests

Experimental Research

When testing a hypothesis, experimental research is the most often used.

Program objectives

They sould also match the needs, sample size, and resources.

1.  Measurement

2.  Use of a design

3.  Analysis of the data

Identifying the effects(outcome) as being attributable to the program and not to other factors related to the evaluation design.

OR

Control for all influences between the groups being compared in an experiment, except for the experimental group.

OR

Degree to which change that was measured can be attributed to the program under investigation.

The health educators ability to generalize the findings of a program.

or

Extent to which the program can be expected to produce similar effects in other populations.

1.  Determine if program goals and objectives are met

2.  Assess effectiveness of organizations, services, or programs.

3.  Identify what changes have occured

4.  Identify what led to changes

Aims to describe the group being studied.

or

Data used to organize, summarize, and describe characteristics of a group.

Gains knowledge about the sample that can be generalized to a similar population.

OR

Data used to determine relationships and causality in order to make genralizations or inferences about a population based on findings from a sample.

1.  Introduction

2.  Literature review

3.  Methodology

4.  Results

5.  Conclusion, Recommendations, or Summary

1.  Front matter- Title of the program, names of the evaluators or researchers, and date of the report

2.  Executive summary

3.  Explanation of the program's background

4.  Problem addressed by the program

1.  Explanation of relevant studies

2.  Background for the study

3.  Relate to the purpose of the study, hypothesis, and target population

4.  Theoretical orientation

1.  Describes how evaluation or research plan was done

2.  Overview of procedures, subjects, and data-gathering insturments

3.  Describtion of the data analsis plan

1.  Evidence tested against teh stated hypotheses or research question

2.  Presents the findings

3.  Discussion of the findings

1.  Conclusion- if analysis supports the hypothesis.

2.  Recommendations- for future research and new research questions.

3.  Summary- restates problems, procedures, and principle findings

Compare and contrast formative and summative evaluation.

(The terms process evaluation and formative evaluation are used interchangeabley and are usually synonymous)

Formative is a process evaluation and summative is impact and outcome evaluation done once the program is completed.

Formative evaluaiot relates to quality assessment and program improvement.  Summative evaluation measures program effectiveness.

1. Convenience

2. Replication

3. Adjustment of variables

4. Establishment of cause and effect relationships

1. Cost
2. Inability to generalize results
3. Securing cooperation

1. Assessing and improving quality
2. Determining effectiveness

In basic sense, programs are evaluated to gain information and make decisions.

1.  To determine achievement of objectives related to improved health status.

2.  To improve program implementation.

3.  To provide accountability to funders, community, and other stakeholders.

4.  to increase community support for initiatives.

5.  To contribute to the scientific base for cimmunity public helath interventions.

6.  To inform policy decisions.

1.  Mandate (policies, statutes, laws) of regulating agencies  ex. % of children immunized for school

2.  Health status of the priority population ex. Rates of mortality & morbidity

3.  Values expressed in the local community ex. Type of school curriculum expected

4.  Satndards advocated by professional organizations ex. Passing scores, cetifications, or registration examinations

5.  Norms established by research ex. Treadmill tests or % body fat

6.  Norms established by evaluation of previous programs ex. Smoking cessation rates or weight loss expectations

7.  Comparison or control groups ex. Used in experimental or quasi-experimental studies

1.  Engage stakeholders

2.  Describe the program

3.  Focus the evaluation design

4.  Gather credible evidence

5.  Justify conclusions

6.  Ensure use and share lessons learned

1.  Those involved in the program operations

2.  Those served or affected by the program

3.  The primary users of the evaluation results

1. Utility standards ensure that information needs of evalation users are satisfied.

2.  Feasibility standards ensure that the evaluation is viable and pragmatic.

3.  Propriety standars ensure taht the evaluation is ethical.

4.  Accuracy standards ensure that the evaluation produces findings that are considered correct.

1.  Planners either fail to build evaluation in the program planning process or do so too late.

2.  Resources may not be available to conduct an appropriate evaluation.

3.  Organizational restructions on hiring consultants and contractors.

4.  Effects are often hard to detect because changes are sometimes small, come slowly, or do not last.

5.  Length of time allotted for the program and its evaluation.

6.  Restrictions that limit the collection of data form those in the priority population.

7.  It is sometimes difficult to distinguish between cause and effect.

8.  It is difficult to separate the effeects of multistrategy interventions, or isolating program effects on the priority population from real world situations.

9.  Conflicts can arise between professional standards and do-it-yourself attitudes with regard to appropriate evaluation design.

10.  Sometives people's motives get in the way.

11. Stakeholders' perceptions of the evaluation's value.

12.  Intervention strategies are sometimes not delivered as inteded, or are not culturally specific.

1.  Being more familiar with the organization and the program history.

2.  Knowing the decision-making style of those in the organization.

3.  Being present to remind others of results now and in the future

4.  Being able to communicate technical results more frequently and clearly.

1.  Bias or conflict of interest

2.  Lack of objectivity

1.  Can often provide a more objective review and a fresh perspective.

2.  Can help to ensure an unbiased evaluation outcome.

3.  Brings a global knowledge of evaluatation having worked in a variety of settings.

4.  Typically brings more breadth and depth of technical expertise.

1.  Isolated, lack knowledge and experience of the program.

2.  Costs more money

Quantitative is deductive in nature (applying a generally accepted principle to an individual case), so that the evaluation produces numeric data, such as counts, ratings, scores, or classifications.  This method is good for well defined and programs.  It compeares outcomes of programs with those of other groups or the general population.  It is used most for evaluation designs. It measures level of occurence, provide proof, and measure levels of actions and trends.

Qualitative is an inductive method (individual cases are studied to formulate a general principle), and produces narrative data, such as descriptions.  It is best used for programs that emphasize individual outcomes or in cases where other descroptive information from participants is needed.  It provides depth of understandding, study motivation, enavel discovery, are exploratory and interpritive, and allow insights ointo behavior and trends.

Experimental design offers the greatest contro over the various factors tahtmay influence the results.  It used random assingment to experimental and control groups with measurement of both groups.  It produces the most interpretable and supportive evidence of program effectiveness.

Quasi-experimental design results in interpretable and supportive evidence of program effectiveness, but usually cannot control for all factors that affect the validity of the results.  There is no random assignment to groups, and comparisions are made on experimental and comparison groups.

Nonexperimental design without teh use of a comparison or control group, thas little control over the factors that affect the validity of the results.

History- event happens between the pretest and postest not part of the program. i.e. nataion nonsmoking campaign in the middle of state campaign.

Maturation- growing older, stronger or wiser between pre & post tests.

Testing- familiarity with the test

Instrumentation- change in measurement of the test.

Statistical regression- high or low scores on the pretest are closer to the mean or average scores on the posttest.

Selection- difference in experimental and comparison groups, lack of randomization.

Mortality (attrition)- drop outs

Diffusion or imitation of treatments- control group interacts and learns from the experimental group.

compensatory equalization or treatments- when services or program is not available to the control group.  They complain and want the program.

Compensatory rivalry- Control group seen as an underdog therefore being motivated to work harder.