How should an evaluation plan be organized: Methods

The Methods section of the evaluation plan spells out the overall evaluation design and data collection strategies to be employed. There are scores of designs and many more data collection strategies that can be used. Unfortunately, traditional evaluation textbooks do not provide sufficient practical guidance in the area of methodology because the examples they commonly include are based upon the assumption that one design will suffice (e.g., a quasi-experimental design that could be used to compare a digital library with a traditional one). If money and time were unlimited, it might be possible to carry out large-scale experimental evaluations, but this is rarely the case. Instead, you will be lucky if you can use several different smaller-scale methods such as usability testing, expert review, and user surveys to collect the information needed to answer your evaluation questions and ultimately inform the decision-making process. Here is a brief example of a Methods section:

The Methods section of an evaluation plan should answer the following questions:

One way to illustrate how your evaluation methods align with your questions is to use a matrix. The matrix below illustrates the relationship between specific questions and the data collection methods used in the evaluation of the graphical user interface (GUI) of a hypothetical digital library. On one axis of the matrix are listed the abbreviated versions of the questions to be addressed by the evaluation. Listed on the other axis are the appropriate data collection methods (i.e., reliable, valid, and feasible) for this particular evaluation. An advantage of using a matrix is that you, your colleagues, your clients, and other stakeholders can review the alignment between the evaluation questions and the proposed methods of collecting data. It also allows you to ensure that each question is addressed by one or more data collection methods. Although it is not always feasible in every evaluation, it is desirable to triangulate most questions with more than one evaluation method.

This sample matrix is by no means an exhaustive list of all the evaluation data collection methods that could be employed in such an evaluation. Other chapters in this Guide provide examples of additional methods.