Why evaluate
Planning
Decisions
Questions
Methods
Organization
Intro
Bkgd
Stakehldrs
Purposes
Decisions
Questions
Methods
Sample
Instrument.
Limitations
Logistics
Budget
What's next
Service eval
Usability eval
Info retrieval
Bibliometrics
Log analysis
Surveys
Interviews
Observations
Experiments
Reporting
References
Credits
Search
PDF version

Instrumentation

The Instrumentation section of the evaluation plan describes the measurement tools to be used in the evaluation. Copies of the instruments can be included in appendices for review by your clients or others. The descriptions in this section should provide enough information to permit readers to judge the various purposes and uses of instruments such as questionnaires, interview protocols, and observation recording tools. Some digital library evaluations will require the development of new instruments, in which case the plan may only include an outline of how the instruments will be developed. Here is a brief example of an Instrumentation section:

INSTRUMENTATION:

  1. A User Questionnaire will pop up on the screen after someone has interacted with DLEE for more than 30 minutes. A copy of the questionnaire can be found in the appendices. This type of pop-up questionnaire has been utilized in previous evaluations of digital libraries conducted by Bill Biggs and Tracey Toliver. Previous evaluations have yielded acceptable support for the reliability and validity of pop-up questionnaires. As illustrated in the appendices, the pop-up survey is very brief with only four questions, but it includes an invitation to link to a longer questionnaire. Previous evaluations have found that 45% of the users complete the pop-up survey, and that 35% of those go on to complete the longer survey.
  2. Usability testing of DLEE will be conducted using a sample of engineering education faculty from North Island University . The protocol for the usability testing can be found in the appendices along with reliability and validity data. The actual testing will be conducted by two experts from Usability Gurus, Inc.

The Instrumentation section of an evaluation plan should answer the following questions:

  • What measurement instruments will be used to collect data for the evaluation?
  • What is the reliability and validity of these instruments?

Regardless of the types of instruments you use, issues of reliability and validity are important . The reliability and validity of instruments must be considered in light of the purposes of the evaluation (Patton , 1997). Reliability deals with the consistency of measurement of an instrument . For example, a bathroom scale that provides the same weight if you step on it ten times in a row can be said to be reliable. Validity is about the degree to which an instrument achieves its aims. For example, if you want an accurate report of your weight, the reliable bathroom scale must be calibrated with another scale of recognized accuracy. It could be giving you the same weight ten times in a row, but be off by five pounds. Any evaluator can learn the fundamentals of establishing the reliability and validity of evaluation instruments, but it may be necessary to hire measurement specialists to provide expert consultation in this area, especially when new instrumentation is being developed.