Why evaluate
Planning
Service eval
Usability eval
Info retrieval
Bibliometrics
Log analysis
Surveys
Interviews
Observations
Experiments
Reporting
What is it
Technique
Samples
Refs & tools
References
Credits
Search
PDF version

How do you prepare evaluation reports?

The reality is that most evaluations are still reported as written documents, although they are often shared electronically as Adobe Portable Document Format ( pdf) files or in other digital formats. A final written report should contain all the elements that will make it useful to the decision makers and other stakeholders. Here is an outline of a typical evaluation report:

  1. Title Page
  2. Table of Contents
  3. Executive Summary
  4. Overview and Background (what was evaluated and purpose of the evaluation)
  5. Decisions (intended to be influenced by the evaluation) and Questions (that were addressed)
  6. Methodology (the evaluation design and any instruments that were used)
  7. Results (organized by methods, e.g. interviews, questionnaires, observations, or by questions)
  8. Discussion and Recommendations
  9. References
  10. Appendices

Most reports start with an executive summary summarizing the findings and presenting the recommendations along with a brief rationale for each recommendation. Pay special attention to crafting a compelling executive summary because this is the only part of your report that many, if not most, decision makers will read. As illustrated in the following hypothetical example, the structure of the executive summary should emphasize the major recommendations stemming from the evaluation and provides minimal explanation wherever required.

Evaluation of the Southern Botanicals Digital Library

Introduction

The Southern Botanicals Digital Libraries (SBDL) is a digital repository of educational resources focused on enhancing K-12 science education through the study of endangered plants. In response to a mandate from the SBDL funding agency, the American Botanical Foundation, to evaluate its efficacy, an external evaluation was conducted by evaluators from Old South University using three primary methods: (1) transaction log analysis, (2) interviews with the teachers and students from selected rural, suburban, and urban school districts, and (3) focus group s conducted with curriculum directors at the annual meeting of the American Science Teachers Society (ASTS).

Overview of Results

The SBDL users download more than 9,000 resources per month. Transaction log analysis indicated that 30 percent of users come from the K-12 community, 25 percent from higher education institutions, 20 percent from other botanical sites, and the rest from the general public or undefined. Just over 80 percent of the K-12 users come from schools in the eleven southeastern states represented in the SBDL collection. Interviews revealed enthusiastic adoption of the SBDL resources in rural and suburban school districts, but minimal usage in large inner-city urban districts. Focus groups at the ASTS meeting indicated that more than half of the science curriculum developers were unaware of the SBDL. Focus group participants praised the diversity and media components of the SBDL, but complained that there were insufficient capabilities for searching for resources that met specific national, state, and district science education goals and objectives.

Recommendations

  • It is recommended that the SBDL strive to incorporate more resources that permit the integration of resources in urban settings. Educators from urban districts are unable to participate in several of the well-received initiatives of the SBDL such as the planting of bog gardens in coordination with the Southeastern Wildlife Organization. Alternative urban gardening projects should be defined and appropriate educational resources should be adopted or created.
  • It is recommended that all educational resources be searchable by national science standards as well as by the standards of the eleven states represented in the SBDL collection. A mechanism whereby school districts might automate the process of linking their science education objectives with the SBDL collection should also be explored.
  • It is recommended that the developers of SBDL seek to become a collection included in the National Science Digital Library (NSDL). This will increase awareness of the SBDL.

In addition, most evaluation reports include appendices that provide greater detail about various aspects of the evaluation. Appendices often include copies of the instruments used in the evaluation and even transcripts of original source data.

Given the nature of digital libraries, consider utilizing alternative reporting formats such as Web pages and video to present your results in the most compelling way. A well-designed Web report would include links to the library itself and to specific features of the library that have been evaluated. An online report can be easily linked to online discussion forums to allow all stakeholders to participate in on-going discussions of the evaluation results. Such discussions can be especially powerful in helping the results of an evaluation to be transformed into action.

Video evaluation reports may require additional resources, but a professional quality video report can have an enormous impact on decision makers. Videos can also be used to kick-off focus group discussions of evaluation reports involving critical groups of stakeholders. Although they are not focused on digital libraries per se, the video reports of educational technology integration initiatives produced by the George Lucas Foundation (http://www.glef.org) provide excellent models for video evaluation reports of digital library projects.