Style Sheet for Evaluation Critique
due October 10

For this assignment, you may choose an evaluation report to critique.  Some sources of evaluation reports are state legislature evaluation units (see, for example, http://legisweb.state.wy.us/progeval/topics.htm), universities (see, for example, http://www.ksg.harvard.edu/pepg/index.htm), journals such as Evaluation Quarterly and Evaluation Review* and many other sites (see, for example http://www.19thcircuitcourt.state.il.us/bkshelf/ddc4_exec/ddc4exec_sum00.htm).  I would like you to show me the report you intend to critique in advance, however, so that I can ensure it has enough information to do so.  If you cannot find a report to critique, I will provide you with one.

Your paper should take the form of a memo written to an agency head who has asked you to critique the evaluation.  Assume that the agency head has not read the evaluation report so that, in the course of your critique, you will need to explain the problem the program was intended to address, the evaluation design, etc.  Your critique should address the following questions:

q       Does the report clearly describe the problem the program is intended to address?

q       Does the evaluation design pose the right evaluation question(s) for study?

q       Were the methodological strategies developed to answer that question adequate and feasible?  What are some of the strengths and weaknesses of those strategies?

q       What method was used for identifying the counterfactual?

q       Did the evaluator assess both process and impact in a meaningful way?

q       Were the data needed to answer the evaluation question available?  How would you assess the process used to collect these data?

q       Is there any reason to suspect the reliability or validity of the data or measures used by the evaluators?

q       Does the evaluation generalize from a specific situation or sample to a larger situation or population?  Is such generalization supportable, given the methodology employed?

q       Are the conclusions reached by the evaluators reasonable and convincing (e.g., is sufficient, credible evidence presented to support them)?

q       In retrospect, how, in your opinion, could this evaluation have been improved?

In answering these questions, your paper should be both descriptive (i.e., explain what was done in the evaluation) and analytical (i.e., comment on the appropriateness, strengths, limitations, potential pitfalls, etc. of the evaluation in each of these respects).  Your paper should be written in narrative form; not simply as answers to the above questions.  It should begin with a brief introduction, and end with a short summary/conclusion.  The use of subtitles is encouraged to make the organization of your paper clear.  It should be no more than five pages, double spaced.  Your grade will be based on how clearly, thoroughly, analytically and succinctly you address the above questions.