Why evaluate
Planning
Service eval
Usability eval
Info retrieval
Bibliometrics
Log analysis
Surveys
Interviews
Observations
What are they
Technique
Case study
Refs & tools
Experiments
Reporting
References
Credits
Search
PDF version

Observations case study

A good example of the application of observations in a digital library context is the evaluation of the Perseus Digital Library (Marchionini, 2000). The Perseus Digital Library (PDL) (http://www.perseus.tufts.edu) is dedicated to providing digital resources for the Humanities. It has been under continuous development since 1987, and represents one of the rare digital libraries that have been intensively evaluated. Data collection methods used in the evaluation of the PDL included observations, interviews, document analysis, and learning analysis. Marchionini describes five types of observational methods used in the PDL evaluation: (a) baseline, (b) structured, (c) participant, (d) think-aloud, and (e) automatic screen journaling.

  • Baseline observations were semi-structured and consisted of the observers sitting in classrooms or labs taking notes of the ongoing activities. The purpose of conducting baseline observations was to help the evaluators become situated within the setting and build relationships with those individuals to be observed.
  • Structured observations involved systematically observing behavior and recording notes for a selected sample of students in a classroom or lab setting. The observations followed an established protocol, for example, the evaluator selected five students and alternated observations between the five students every three minutes. In the context the PDL evaluation, details such as whether students were taking notes or looking at the instructor, were recorded. Specifics of what details to record in any observation will depend on the setting and specified evaluation goals.
  • Participant observations involved sessions in which the evaluator was allowed to interact (ask or answer questions) with the students being observed. All sessions were audio-taped, and evaluators had a semi-structured protocol to guide their interactions.
  • Think-aloud observations as used in the PDL evaluation were much like the think-aloud protocols described in Chapter 4 – Usability Evaluation of this guide. In Marchionini's evaluation of the PDL, students were audio-taped and asked to think aloud while working on various tasks.
  • Automatic screen journaling described in the PDL evaluation was akin to transaction log analysis as described in Chapter 7 of this guide. Data collected from the automatic screen journaling was used to determine interaction patterns such as the number of requests for different resources and temporal patterns of access.

The use of these five observational data collection methods, combined with the other data collection methods such as document analysis, interviews, and learning analysis allowed Marchionini and his team of evaluators of Perseus to gather quality information to guide decision about refining and extending this notable digital library. A number of important recommendations for the PDL as well as for digital libraries in general have been made on the basis of the findings and interpretations stemming from this comprehensive evaluation.