PA 784
Fall 2002
Due Oct. 3

Homework 3

Evaluation Design

Please choose any one of the following evaluation designs and:

1. Identify the type of design that is proposed (see Mohr, table 4).

2. Suggest which, if any, of the following threats to internal validity are likely to be an issue with the design, for what reason(s), and what, if anything, could be done to nullify or minimize the bias.

3.  Comment on what might undermine the  external validity of the design, if anything and how external validity might be strengthened, if at all..

Designs

1. Smoking Cessation

Patients under care by UCSF physicians who are identified from their records as smokers are randomly assigned to an experimental and control group.  On their next visit, those in the experimental group are asked to watch a short video about the dangers of smoking.  One month following their next visit, all of those in the experimental and control group are contacted and asked a series of questions about their knowledge of smoking-related illnesses and whether they are still smoking. The evaluation compares the knowledge of each group about smoking related illnesses and the number in each group that has quit smoking.

2. 360 Degree Performance Appraisal

The program to be evaluated is a 360-degree performance appraisal. (This is where performance information about an employee is gathered from several sources, such as peers, supervisors, customers, subordinates, rather than just from the employee’s supervisor, as is traditionally the case).   Employees in two City government departments were assigned to the experimental group while employees in two other departments comprised the control group.

Employees in the experimental group completed a questionnaire asking them three categories of questions:  (1) their comfort with having co-workers rate their performance; (2) their comfort with their supervisors’ rating their performance; and (3) their own comfort in rating their own performance.  They then received a 1-hour orientation to the 360-degree system.  Over the next six months the 360-degree appraisals were administered. Then the employees were asked to complete the same questionnaire again.

The control group completed the same questionnaire during the same two time periods the experimental group did, but were not informed about the new 360-degree system nor that it was being pilot tested in the experimental group.  The evaluation compares the average scores of the experimental and control groups on the three dimensions (categories) described above.

 3. Increasing Faculty Productivity

The grant program is intended to increase the productivity of Biology and Chemistry faculty by providing funding for equipment, supplies and travel to conferences.  Funding is offered to any faculty in the Biology and Chemistry departments who want to participate.  “Productivity” is defined as the number of conference papers presented and publications in scholarly journals.  An initial count of such papers and publications is taken at the end of the spring 1997 semester.  Funding is available at the beginning of the fall 1997 semester.  At the end of the Spring 2000 semester, another count of papers and publications is taken.  The evaluation compares the average number of papers and publications produced by faculty per year prior to the Fall of 1997 with the average number produced per year during the funding period.

 4. Encouraging Safe Sex

A program was designed to teach high school students about the danger of sexually transmitted diseases (STD’s) and the importance of safe sex.  City high schools were asked to volunteer for the project.  Within each high school that volunteered, homeroom classes of seniors were randomly assigned to “treatment” or “control” groups.  Classes in the first category were shown an hour film on the topic while classes in the second category were not.  One week later, all of the classes were given a test with questions about STDs and the safety of sexual practices.  The evaluation consists of comparing the average score of the two groups on these tests.