What kinds of methods should you use?

The delineation of important decisions and unambiguous questions is essential before deciding upon the methods of evaluation. Unfortunately, too many evaluations start with the specification of methods. People think: “We have to do an evaluation. Let's begin by designing a survey.” Survey methods are just one option for evaluation methods, and this Guide has been written to provide you with a better understanding of how to choose the most appropriate methods for answering your evaluation questions and ultimately informing decision making.

It helps to think of evaluation methods as tools. Just as you would not select a carpentry tool (hammer, saw, or plane) before understanding the nature of the task you need to accomplish, you should not choose evaluation methods until you are as clear as possible about the questions you need to answer in order to inform the decision-making process. There are numerous evaluation methods (e.g., usability testing) and even more specific data collection strategies (e.g., keystroke analysis) that can be used within any given method. One key to successful evaluation is matching these options to the decisions and questions of your stakeholders while adhering to the budget and timeline limitations of your situation.

Most evaluations will demand multiple methods (Mark & Shotland, 1987). You will often need to “triangulate ” your findings. You can triangulate findings by using more than one method to collect data related to an evaluation question. For example, suppose you are trying to decide whether the search functions in your digital library should only search across resources in the collections that are part of your library, or allow searching across the entire World Wide Web (WWW). A question you might address to inform this decision could be “What are middle school teachers' attitudes toward the use of open searching on the Internet?” An emailed questionnaire designed to elicit teachers' views about Internet searching would be one way of collecting that data, but most people, including teachers, are turned off by questionnaires and wary of sharing information via email. Thus, they may not provide you with sufficiently detailed information about their real opinions about this matter. A better strategy might be:

• conduct a series of focus groups with teachers, administrators, and parents about the pros and cons of open searching by middle school students,
• review the policies established by a representative sample of school districts concerning Internet access by students, and
• review the professional library literature concerning recommendations for Internet search policies and procedures.

Most carpentry jobs require multiple tools, and similarly, most evaluations require multiple methods. There are many examples of library evaluations that have employed multiple methods. For example, Norlin (2000) employed surveys, unobtrusive observations, and focus groups to evaluate user services in a university library.