THE ROLE OF PROCESS EVALUATION
IN THE TRAINING OF FACILITATORS FOR AN ADOLESCENT HEALTH EDUCATION PROGRAM
Journal
of School Health
April 2000
vol 70, Issue #4, p. 141
ABSTRACT: This article reports on the process evaluation of the training of facilitators for the Adolescent Social Action Program, a health education program in Albuquerque, New Mexico that trained college students and adult volunteers to work with middle school students. From the process evaluation data collected throughout a four-year period (1995-1998), data relevant to training are described: facilitator characteristics, facilitator training, curriculum implementation, and use of the program's model designed to promote critical thinking and dialogue. Results indicated that, though most facilitators reported the training was sufficient to enable them to implement the curriculum, they did not completely do so, especially as groups reached their final sessions. Facilitators covered the core curriculum content, but often failed to follow through with the more abstract activities. The need to perform and report the process evaluation in time to provide ample opportunity for trainers and curriculum designers to make appropriate adjustments is discussed. (J Sch Health. 2000;70(4): 141-147)
Many health programs use community members as lay health workers to implement systematic interventions developed by professionals. These community members, including lay health promoters, school teachers, college students, and youth, are trained to serve as facilitators and paraprofessionals. Training for such programs is a crucial element in the success of intervention implementation, but process evaluation literature has not adequately addressed training. Among the many benefits, process evaluation can reduce the likelihood of type III error: the mistaken assumption that a program was implemented as planned,(n1) as well as identify specific components that were implemented poorly or not at all. If used in a timely manner, process evaluation can provide important feedback to the trainer and curriculum developer, and can ultimately affect the interpretation of outcome evaluation.
This article reports on the process evaluation of a primary prevention health program that relies on lay paraprofessionals to implement the program. Aiming to prevent high-risk behaviors among adolescents, the Adolescent Social Action Program (ASAP) trained college students and adult volunteers to work as facilitators of adolescent groups. From the numerous process evaluation measures used, this paper focuses on those measures specifically related to the training of facilitators. In examining this component of the process evaluation, the paper presents the important benefits of monitoring implementation and providing early feedback to the training process.
LITERATURE REVIEW
Process evaluation is the key to understanding the internal dynamics of an intervention trial and to quality control. Israel et al(n2) list the objectives of process evaluation as "feedback on program implementation, site response, participant response, practitioner response, and personnel competency." Process evaluation also can be used as a formative tool, to improve and refine interventions while being implemented, as well as explain and interpret intervention outcomes.(n2-n4) Furthermore, process evaluation can provide the critical documentation necessary for sustaining and replicating successful community-based trials.(n5, n6)
As with other types of evaluation, a credible process evaluation is theoretically based and methodologically sound. Israel et al,(n2) Helitzer et al,(n4) Chen,(n7) and Weiss(n8, n9) promote expression of the theoretical premise of the intervention; that is, outlining the planned sequence of steps of the intervention that theoretically leads to the desired outcome. From this outline, process evaluation is developed, and the most salient indicators to measure implementation are identified. Rossi and Freeman(n10) advocate substantive and comprehensive process evaluation which includes a number of quantitative and qualitative data sources. Use of multiple methods increases the range of data collected, promotes data and methodological triangulation, and increases the validity of conclusions. They suggest use of the following: 1) direct observation by the evaluator; 2) written records; 3) data from delivery agents of the intervention; and 4) data from intervention trial participants.
Process evaluations have been reported more frequently in recent literature. Between 1994-1999, at least nine reports were published of process evaluations conducted on a variety of public health interventions: an obesity prevention trial for Native American school children (PATHWAYS);(n4) high blood pressure control programs;(n11) health promotion programs for breast and cervical cancer;(n12) a cancer-related risk reduction community project;(n13) a child and adolescent trial for cardiovascular health (CATCH);(n3) alcohol and drug abuse prevention programs;(n14, n15) tobacco use prevention interventions;(n16-n18) and a community health promotion project.(n19) Examples of research on training in the health field include studies examining: the best training media to implement health curricula, eg, live workshops vs. video training;(n18) adults vs. peer trainers delivering resuscitation training;(n20) and program implementation aimed at high-risk youth.(n14)
Training evaluations are useful in providing feedback for decision-making about training content, curriculum, and strategies, as well as in determining the marketability of the training.(n21) Training evaluators should pay particular attention to whether transference of training, "the degree to which trainees effectively apply the knowledge, skills and attitudes gained in a training context,"(n21) has successfully occurred. Since training outcomes depend on numerous factors (eg, personal motivation, learner characteristics, learning process, amount and method of training and trainer/supervisor attitude), several methods should be used to assure the fidelity of implementation by trainees.(n18, n21-n23)
PROGRAM DESCRIPTION
The Adolescent Social Action Program (ASAP), a primary prevention program based at the University of New Mexico School of Medicine, used trained university students and community members to address risky health behaviors among adolescents. Since 1984, ASAP collaborated with the University Hospital, the county detention center, and more than 30 urban and rural multiethnic schools to meet its dual goals of encouraging youth to make healthier decisions and of empowering them to be involved in social action to improve the health of their communities. ASAP targeted youth in seventh grade, as a strategic time to prevent or delay the normal doubling of alcohol experimentation and use from seventh to ninth grade.(n24) Previous evaluation efforts of some Albuquerque public schools that received the ASAP program suggested treatment effects, but the National Institute on Alcohol Abuse and Alcoholism (NIAAA) grant was the first opportunity for systematic process and outcome evaluation research.
The core of the ASAP curriculum was a six-week intensive experience during which small groups of youth (2-9 students) were brought into the hospital and detention center to interact with patients and jail residents with problems related to alcohol abuse, tobacco use, drugs, interpersonal violence, and other risky behaviors.(n25, n26) ASAP facilitators directed the youth groups in Freirian structured dialogue about the patient stories, elicited health and social issues important to the youth, and led the groups through a systematic curriculum of decision-making, communication, conflict mediation, problem-posing, and resistance to peer pressure skills. Specifically, the curriculum consisted of six sessions: one team-building session, three separate hospital visits, one detention center visit, and one community action session. An orientation for parents was also held for each student group before the onset of the six-week curriculum. Under the NIAAA grant, 63 groups of students received the curriculum during the course of seven semesters.
ASAP facilitator training was offered in two distinct parts. The first was an intensive 16-hour (over the course of two days) training at the beginning of each semester, which offered an introduction to ASAP theory and program, practice of the seven-session curriculum, a hospital tour, and discussion of facilitator roles and responsibilities. Second, facilitators attended a one-hour, academic seminar held once a week for the duration of one semester (13 weeks). The first 16-hour training was mandatory, whereas attendance at the weekly seminar was only enforced for facilitators taking college credit.
PLANNING THE EVALUATION
The process evaluation for the ASAP program was comprehensive, incorporating measures of competing programs at the school level; facilitator characteristics and experience; student participation, exposure, and reaction to the program; student participation in other programs; student perceptions of barriers to and successes of the program; written reports about the social action projects; facilitator documentation of session content and perceptions of problems, difficulties, and barriers; facilitator training evaluation; and observer documentation of session content. Figure 1 illustrates the process evaluation model. The various measures were collected through a variety of instruments and methods, both qualitative and quantitative. Data for this article came from the four sources highlighted in the model: 1) self-administered questionnaires that included facilitator characteristics, 2) training evaluation forms completed by facilitators, 3) facilitator check-off lists that documented curriculum implementation, and 4) observations of group sessions.
Facilitator Questionnaires
Each facilitator was asked to complete a questionnaire requesting background information (eg, age, level in school, ethnicity, and experience with adolescents), prior experience with and attitudes toward the curriculum topics (eg, alcohol and drug use), and their perceived self-efficacy about influencing others. Analysis of background data provided a description of the facilitator population.
Training Evaluation
After each two-day training session, facilitators were asked to fill in a post-training evaluation form. These evaluation forms were administered five of the seven semesters under the NIAAA grant by the ASAP trainer. Since facilitators were required to attend the two-day training only once, all evaluation forms were completed by first-time facilitators. This evaluation instrument addressed training content; training staff competence; participants' grasp of ASAP goals and the theory; and perceptions of training materials and activities. Question #4 was analyzed: "This training enabled me to feel I could implement the ASAP Program" (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree).
Facilitator Check-Off List
Reflecting the curriculum, this form contained lists of content/activity areas that should have been covered in the five sessions where the paired facilitators worked alone with the students, minus the sixth session during which families and friends participate. The team-building session was labeled "Session I;" the detention center visit was "Session IV;" and the three hospital sessions were "Sessions II, III, and V." For each student group, the two cofacilitators were instructed to fill out one form (0 = didn't cover, 1 = touched on briefly, 2 = mostly covered, 3 = completely covered). Data from these lists were compiled by adding the scores for all 63 groups for each content area to determine the total coverage of each activity. Implementation of the curriculum also was analyzed by session, by semester, and over all semesters.
Observation of Session Content
The observation form contained three sections: questions on "Student Behavior," "Facilitator Behavior," and "Content." Research assistants were trained to observe Session V. Student behaviors under observation included demonstration of: belief in group action; sense of belonging; self-efficacy; sense of responsibility; ability to articulate social context and risky behavior; group process; and participation, cohesion, attention and discussion of concepts such as culture, community, and stereotype. Facilitator behaviors observed included: demonstration of the use of dialogue to move students to critical thinking and eventually social action; and facilitator group process skills (eg, domination, respect, impatience, question-answering, interaction, didactic skills, probing, and discussion generation). A total of 29 groups was observed.
The "Content" section was analyzed to compare observer and facilitator reports of implementation. This section of the observation form consisted of the same list of content areas and rating scale as the facilitator check-off list (see previous section). One additional content area was added to the observer list, the use of the SHOWED model (Figure 2) which promotes the critical thinking component of the curriculum. In order to triangulate these data, two other questions were posed: "facilitator leads a dialogue based on the SHOWED model" ("yes" or "no"), and "students use the SHOWED model in analyzing problems" (3-all of the model, 2 = most of the model, 1 = some of the model, and 0 = none of the model).
FINDINGS FROM THE EVALUATION
Facilitator Background Information
Over the course of seven semesters (under the NIAAA grant), 98 facilitators completed the ASAP facilitation training course. Table 1 contains data on gender, age, ethnicity, marital status, education, prior experience working with adolescents, parenting adolescents, and number of semesters with ASAP.
Facilitator Confidence in Training
Of the 46 evaluation forms collected, 37% strongly agreed and 57% agreed (totaling 94%) that the training enabled them to feel they could implement the ASAP curriculum. Four percent indicated "don't know/neutral" and 2% disagreed.
Implementation Reported by Facilitators
Check-off lists from all 63 groups were collected and analyzed. As an example, Figure 3 shows coverage of the content areas for Session V. Some activities were covered more completely ("Advertising Activity," "Patient Interviews," and "Student Comment Sheets") than others ("Discuss Students' Communities" and "'But Why?' Root Cause Analysis").
Of the 34 total content areas (activities) from all five sessions, 15 content areas were highly covered (at least 85%). Another 15 of the content areas were covered 51%84%, while four were poorly covered (50% or less).
Coverage was then analyzed by semester. Table 2 contains the average scores for each session by semester, with the total average score for all five sessions by semester. Since the number of content/activity areas in each session varied (5-9), these average scores were then translated to percentage of curriculum covered. For example, the total average scores (Sessions I-V) by groups in the first semester varied from a low of 66 to a high of 88 (65%86%), with an average score of 76.2 out of a total possible score of 102 (75%). (The low score of 28 in the second semester was from a group that canceled a session.)
When these data were examined by session (for all 63 groups), more variability appeared than by semester (last row of Table 2). The decline in coverage revealed that the initial sessions (I and II) and the jail session (IV) were better covered than later sessions (III and V).
Comparing Observer and Facilitator Implementation Scores
Figure 4 contains the scores for the coverage of content areas (for Session V) given by observers as compared to the scores provided by facilitators on the facilitator check-off list. Since 29 groups were observed, the same 29 groups were selected for comparison from the check-off list data. Observers gave lower implementation scores than the facilitators for 19 of 29 groups. Observer and facilitator scores were the same for four groups, and observers gave higher scores than facilitators for six groups.
SHOWED Model
Analysis of data from the "Content" section indicated that groups used the SHOWED model, at least briefly, in 55% of the observed sessions. The model was covered completely in 17% of the sessions, mostly covered in 7%, touched on briefly 31%, but not covered at all in 45% of the sessions. These results agreed with the analysis of the second indicator from the "facilitator behavior" section; in 59% of the observed sessions facilitators led a dialogue based on the SHOWED model. Analysis of data from the "student behavior" section showed higher usage among students than the facilitators. Students used the model in 72% of observed groups: all of the model in 10% of the sessions, most of the model in 10%, some of the model in 52%, but none of the model in 28% of the sessions. However, if the goal was to completely or mostly cover the model, facilitators in only 24% of the observed sessions and students in only 20% of the sessions met this goal.
IMPLICATIONS OF THE EVALUATION
Curriculum Implementation
Analysis of these data from the process evaluation of the ASAP program during the period of the NIAAA grant (Spring 1995-1998) raised serious programmatic questions, as well as methodological questions related to use of process evaluation. Though facilitators were confident that the training prepared them to implement the ASAP curriculum, results showed inconsistent and lower than desirable implementation. Curriculum implementation appeared to be consistent from semester to semester, suggesting consistency in training, but a decline occurred in implementation as sessions progressed.
Content/activity areas that received high implementation scores include those activities that were more concrete (vs. abstract), repeated in several sessions, expected to be discussed in the seminars, and accompanied by visual materials. For example, the highly implemented patient/inmate interviews were expected to be covered in all of the hospital sessions and were the core of the debriefing discussions during the weekly seminars. Other content areas that scored high included introduction and review content areas. These activities were helpful for them to break the ice, reinforce concepts, and maintain the flow of the curriculum. Poorly-covered content areas (50% or less) tended to be activities that were more abstract, time-consuming, and those that required more skill on the part of the facilitators. For example, though the patient/inmate interviews were well-implemented, facilitators often failed to follow through with the next more difficult and abstract task of using the SHOWED model as a tool for engaging the youth in critical thinking and dialogue.
Another reason that some content/activity areas were executed poorly or not at all might have been a change in the design of the sessions for the NIAAA grant. The four, three-hour sessions (totaling 12 hours) were shortened to five, two-hour sessions (totaling 10 hours) under the NIAAA grant. Though a new session was added, this change did not adequately compensate for the time lost in the other sessions. With the decline of total hours, a few activities were deleted, but more than could be covered remained in the curriculum. This fact might have led facilitators to believe that some activities were optional. Furthermore, as facilitators gained confidence in working with the youth, they may have interpreted greater flexibility in coverage of activities. This action could have been a desired goal, as change agents innovate and make changes to own the curriculum.(n27) However, the result was less standardization in coverage of the curriculum than desired for a research project.
When scores reported by facilitators were compared to scores given by observers, many groups were observed as implementing less than what facilitators reported. However, two serious concerns emerged about the observation. First, results revealed low inter-rater reliability: observers were found to be rating groups differently (ie, one observer may have consistently scored the groups "harder" than other observers). Second, the criteria for scoring (for both observers and facilitators) was not sufficiently spelled out to maximize consistency of rating.
Implications for ASAP Training
Process evaluation can provide information on whether strategically selected program constructs were actually emphasized in the training as intended. For the ASAP program, certain constructs were considered more important than others (eg, self-efficacy, critical thinking, dialogue, and social-skills development), but many of these elements were not implemented in the group sessions. Had ASAP staff and trainers been provided with timely process evaluation feedback, adjustments could have been made to the training and curriculum to address these implementation problems.
Though the process evaluation feedback was neither systematic nor timely during the actual NIAAA grant, there were instances of process evaluation feedback to the ASAP program. For example, before the analysis of data began, observers informed the ASAP trainer that facilitators were often not using the SHOWED model. The trainer attempted to correct this defect by subsequently instituting a quiz for the facilitators on the SHOWED model in the ASAP semester course. By emphasizing this element of the curriculum, the trainer may have been able to increase the implementation level. However, because some faculty providing credit to the college facilitators strongly objected to this unexpected change, the quiz was only used for two semesters, and no other method to increase the use of the SHOWED model was implemented.
The process evaluation results revealed that ASAP training should have emphasized abstract thinking. During the weekly seminars in which the facilitators reported on their sessions, the ASAP trainer could have inquired about each of the activity areas (whether each was executed and how facilitators felt about their ability to implement the activity). Other suggestions for training include critiquing practice facilitation, adding more supervision, and providing additional training midway through the ASAP course.
CONCLUSION
After these data were analyzed, ASAP staff decided that full coverage of all activities was not as important as the ability to detect critical thinking and self-efficacy. Though standardization of curriculum implementation was one of the goals, a more important goal was the quality of the group and empowerment process for the facilitators and youth. To capture this information, the process evaluation measures would have had to look more closely at the theory of the program and make assessments according to the theoretical assumptions of how critical thinking and self-efficacy occur.(n4) The check-off list, as well as the items used from the observations, measured quantity of implementation, and not quality of implementation.(n18) Qualitative methods, such as information on barriers and contributors to implementation from the perspective of facilitators, should have been included.(n10) For example, the significant variability between groups in terms of coverage may have been due to differences in factors beyond the control of the facilitators (eg, unruly students and unavailability of patients). Triangulating the coverage data with qualitative data that provides pertinent information (eg, facilitator logs/journals of sessions) may have explained much of the variability. Furthermore, the measures that were used could have been more standardized. Filling out both the check-off list and observation list required judgment, but there was no clear set of criteria for scoring. Training for session observers and facilitators should have been enhanced, and the criteria for scoring should have been clearly defined.
Programs that rely on community members to implement projects can benefit from evaluating the success of their training. For this evaluation to be helpful, the evaluation must assess what is thought to be crucial to improving outcomes, and data must be analyzed rapidly so information is immediately available to program staff. For supervisors, trainers, and project investigators to benefit, process evaluation needs to 1) follow the program theory, 2) include the project staff in its development, 3) have clear measures and criteria for standardization, and 4) be analyzed quickly. Though process evaluation can be challenging and time-consuming, the insight gained can easily outweigh the costs.
Table 1 Facilitator
Characteristics
Legend for Chart:
A - Gender (n = 87)
B - School Status (n = 89)
C - Parents of Teens (n = 90)
A B C
Female 60% Undergraduate 81% Yes 14%
Male 40% Graduate 19% No 86%
Legend for Chart:
A - Age (n = 87)
B - College (n = 89)
C - Experience with Teens (n = 85)
A B C
20-25 39% Liberal Arts 33% < One year 26%
26-35 40% Education 11% 1-3 years 33%
36-45 14% Nursing 11% 4-6 years 8%
46 and older 7% Medical 3% > 6 years 5%
Other 42%
Legend for Chart:
A - Ethnicity (n = 77)
B - Marital Status (n = 88)
C - Involvement with ASAP (n = 98)
A B C
Caucasian 60% Single 60% 1 semester 82%
Hispanic 30% Married 20% 2 semesters 14%
Native American <1% Divorced 8% 3 semesters 3%
Asian <1% With partner 10% 5 semesters 1%
Table 2 Facilitator
Reported Implementation of ASAP Curriculum by Semester[*]
Legend for Chart:
A - Semester
B - Session I(21)
C - Session II(27)
D - Session III(21)
E - Session IV(15)
F - Session V(18)
G - Total Score (102)
H - Range (0-102)
A
B C D E F G H
1
18.6 21.4 11.4 12.4 12.4 76.2 66-88
(89%) -0.8 (54%) (83%) (68%) (75%) (65-86%)
2
16.3 21.3 15.3 14.4 12.0 79.3 28-92
(77%) (79%) (73%) (96%) (67%) (78%) (28-90%)
3
19.1 21.7 14.2 9.3 10.5 74.8 59-96
(91%) (80%) (68%) (62%) (58%) (73%) (58-94%)
4
18.6 20.1 13.8 11.1 12.5 76.1 66-89
(89%) (74%) (66%) (74%) (69%) (75%) (65-87%)
5
18.4 20.8 14.8 13.7 13.3 81.0 64-94
(88%) (77%) (71%) (91%) (74%) (79%) (63-92%)
6
17.9 26.1 17.0 13.7 14.2 88.9 61-97
(85%) (97%) (81%) (91%) (79%) (87%) (60-95%)
7
17.8 19.5 10.5 11.6 12.1 71.5 62-82
(85%) (72%) (50%) (77%) (67%) (70%) (61-80%)
All semesters
(63 groups)
18.2 21.2 14.1 12.1 12.5 78.1 28-97
(87%) (79%) (67%) (81%) (69%) (77%) (28-95%)
[*] - Scores represent averages for each session. Numbers in
parentheses in the first row show the highest possible score
for that session.
DIAGRAM: Figure 1; Conceptual Model of Process Evaluation for the Adolescent Social Action Program
Figure 2 The SHOWED Model
Used from Paulo Freire in Most of ASAP Sessions to Promote Critical Thinking and Dialogue
S What did students see and observe while talking
to patients?
H What is happening in their stories?
0 How do the patients' stories relate to our lives?
W Why is this a problem for young people?
E What would empower this person or us to change?
D What can we do to improve our lives or the lives
of other young people?
Figure 3 Coverage of
Individual Activities in Session V by all NIAAA Groups from Check-Off Lists
Average Score for All 63 Groups
(Total Possible = 189)
Advertising Activity 170
Discuss Students' Communities 95
Patient Interviews 156
"But Why?" Root Cause Analysis 61
Student Comment Sheets 175
Prepare for Session 6 145
Activity/Content Areas for Session V
GRAPH: Figure 4; Graph of Observer and Facilitator Scores of ASAP Curriculum Implementation
References
(n1.) Steckler A, McLeroy KR, Goodman RM, Bird ST, McCormick L. Toward integrating qualitative and quantitative methods: an introduction. Health Educ Q. 1992;19:1-9.
(n2.) Israel BA, Cummings KM, Dignan MB. Evaluation of health education programs: current assessment and future directions. Health Educ Q. 1995;22:364-389.
(n3.) McGraw SA, Stone EJ, Osganian SK, et al. Design of process evaluation within the Child and Adolescent Trial for Cardiovascular Health (CATCH). Health Educ Q. 1994;2(suppl):5-26.
(n4.) Helitzer DL, Davis SM, Gittelsohn J, et al. Process evaluation in a multisite, primary obesity-prevention trial in American Indian schoolchildren. Am J Clin Nutr. 1999;69(suppl):816-824.
(n5.) Dehar M, Casswell S, Duigan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17:204-220.
(n6.) Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87-108.
(n7.) Chen H. Theory-Driven Evaluations. Newbury Park, Calif: Sage Publications; 1990.
(n8.) Weiss CH. How can theory-based evaluation make greater headway? Eval Rev. 1997;21(4):501-524.
(n9.) Weiss CH. Theory-based evaluation: past, present, and future. New Dir Evaluation. 1997;76:41-55.
(n10.) Rossi PH, Freeman HE. Evaluation: A Systematic Approach. 5th ed. Newbury Park, Calif: Sage Publications; 1993.
(n11.) Roccella. Considerations for evaluating community high blood pressure control programmes. J Hum Hypertens. 1996;1(suppl): 17-19.
(n12.) Scheirer MA, Shediac MC, Cassady CE. Measuring the implementation of health promotion programs: the case of the Breast and Cervical Cancer Program in Maryland. Health Educ Res. 1995;10(1):11-25.
(n13.) van Assema P, Steenbakkers M, Eriksen M, Kok G. The process evaluation of a Dutch community health project. Int Q Commun Health Educ. 1994-5; 15(2): 187-207.
(n14.) Eggert LL, Thompson EA, Herting JR, Nicholas LJ, Dicker BG. Preventing adolescent drug abuse and high school dropout through an intensive school-based social network development program. Am J Health Promo. 1994;8(3):202-215.
(n15.) Scott DM, Merkel PA, Barlow TW. Process evaluation of Nebraska's team training project. J Drug Educ. 1994;24(3):269-279.
(n16.) Elder JP, McGraw SA, Stone EJ, Harsha D, Wambsgans K, Greene T. CATCH: process evaluation of tobacco policies, food services, and secular trends in CATCH. Health Educ Q. 1994;2(suppl): 107-127.
(n17.) Elder JP, Woodruff SI, Sallis JF, de Moor C, Edwards C, Wildey MB. Effects of health facilitator performance and attendance at training sessions on the acquisition of tobacco refusal skills among multi-ethnic, high-risk adolescents. Health Educ Res. 1994;9(2):225-233.
(n18.) Basen-Engquist K, O'Hara-Tompkins N, Lovato CY, Lewis MJ, Parcel GS, Gingiss, P. The effect of two types of teacher training on implementation of Smart Choices: a tobacco prevention curriculum. J Sch Health. 1994;64(8):334-339.
(n19.) Wickizer TM, Von Korff M, Cheadle A, Maeser J, Wagner EH. Activating communities for health promotion: a process evaluation method. Am J Public Health. 1993;83(4):561-567.
(n20.) Lester C, Donnelly P, Weston C. Is peer tutoring beneficial in the context of school resuscitation training? Health Educ Res. 1997;12(3):347-354.
(n21.) Kraiger K. Integrating training research. Training Res J. 1996;1(1):5-16.
(n22.) Gagne RM. Learning processes and instruction. Learning Res J. 1996;1(1):17-28.
(n23.) Quinones MA, Ford JK, Sego DJ, Smith EM. The effects of individual and transfer environment characteristics on the opportunity to perform trained tasks. Training Res J. 1996;1(1):29-48.
(n24.) Drug Free Schools and Communities Program of NM's Children, Youth Families Dept, Risk Reduction Services Division. Albuquerque Public Schools Summary - Tobacco, Alcohol and Drug Survey. Albuquerque, NM: Research & Polling, Inc.; 1994.
(n25.) Wallerstein N, Bernstein E. Empowerment education: Freire's ideas adapted to health education. Health Educ Q. 1988;15:379-394.
(n26.) Wallerstein N, Sanchez-Merki V. Freirian praxis in health education: research results from an adolescent prevention program. Health Educ Res. 1994;9(1):105-118.
(n27.) Hall GE, Hord SM. Change in Schools: Facilitating the Process. Albany, NY: State University of New York Press; 1987.
~~~~~~~~
By Deborah Helitzer; Soo-Jin Yoon; Nina Wallerstein and Lily Dow y Garcia-Velarde
Deborah Helitzer, ScD, Research Assistant Professor/Director, Office of Evaluation; or <helitzer@unm.edu>; Soo-Jin Yoon, MA, Research Scientist, Office of Evaluation; and Nina Wallerstein, DrPH, Associate Professor/Director, Masters in Public Health Program, Dept. of Family and Community Medicine, University of New Mexico, 2400 Tucker NE, Family Practice Center, Room 176, Albuquerque, NM 87131; and Lily Dow y Garcia-Velarde, PhD, Assistant Professor, College of Health and Social Services, Dept. of Health Science, New Mexico Southern University, P.O. Box 30001, Las Cruces, NM 88003. This research was supported by NIAAA Grant #1 RO1 AA08943-01A3. This article was submitted September 8, 1999, and revised and accepted for publication November 22, 1999.