The five basic steps of an evaluation

There are many complex evaluation models in textbooks and scholarly papers, and evaluation specialists often have graduate degrees in this area. However, the evaluation model promoted in this guide involves five basic steps:

• Identify the decisions that you or others involved in your digital library enterprise must make.
• Identify the questions that need to be addressed to inform the pending decisions.
• Identify the evaluation methods and instruments that will be used to collect the information needed to address these questions.
• Carry out the evaluation in a manner that is as effective and efficient as possible.
• Report the evaluation results in an accurate and timely manner so that it can provide the information you and others need to make the best possible decisions.

Sounds simple, doesn't it? It makes you wonder why people haven't conducted evaluations of digital libraries more frequently in the past. We attribute at least part of this failure to evaluate to the fact that people rarely begin evaluation planning by identifying the decisions that an evaluation should inform. Instead, they usually begin an evaluation by struggling to identify the aspects of digital libraries to be evaluated (e.g., user interface or collection quality), the criteria for evaluation (e.g., accuracy or relevance), or the most appropriate methods (e.g., usability testing or online surveys). Beginning with these issues inevitably leads to disagreements and arguments that can stall or even halt the evaluation process. Although identifying decisions up front is not an easy task, the payoff is that all the other issues (questions, methods, instruments, criteria, standards, analysis, reporting, etc.) flow naturally from the specification of the decisions that the evaluation should inform. Chapter 2 in this Guide is devoted to planning an evaluation, including the critical process of identifying decisions.