 |
|
|
|
|
|
|
|
|
 |
  |
 |
 |
 |
|
|
|
|
|
|
How do you do observations?
The value of using trained and skilled observers in your digital library evaluation will be evidenced in the quality of data they gather. For purposes of reliability, most evaluations will involve multiple observers conducting multiple observations. Doing observations takes considerable skill and training, but trained observers can report the same event with accuracy and reliability. Patton (2002) states that to become a skillful observer, training is required in the following areas:
- Paying attention: seeing what there is to see, and hearing what there is to hear
- Writing descriptively
- Discipline in recording field notes
- Separating detail from trivia to achieve notes that are detailed and not overwhelmed by the trivial
- Using rigorous methods to triangulate and validate observations
- Reporting the strengths and limitations of one's own perspective
If you want to use observational methods in the evaluation of your digital library, you probably desire to learn more about how your digital library is implemented in a real-life setting like a classroom or the spaces where students study or collaborate such as coffee shops. For example, suppose you are interested in how your digital library is used in an undergraduate introductory biology class. If so, there are a number of dimensions on which observational approaches vary that can be used to shape the design of the observational approach to be used in the digital library evaluation. Patton (2002) outlines six different dimensions to consider:
Role of the observer |
What role will the observer (evaluator) play in the setting in which the observation is to take place? The observer can act as a full participant in the evaluation context, or be an outside onlooker, unobtrusively making observations. For example, will the evaluator participate in all activities in the biology course like a fully participating student or be a removed observer with no role in the class, or some combination in-between? |
Insider versus outsider perspective |
Will the evaluator approach the observation with the goal of capturing the insider perspective – recording both what it is like to be part of the class as a student, and also what is happening to everyone else in the class? Or will an outsider perspective be taking by the evaluator, focusing on capturing the separate events and their relation to each other, from a distance? |
Who does the observing |
A single evaluator, a team of evaluators, or a combination of evaluators and participants (in the example of the biology class, this would include students and/or the instructor) could gather observational data. |
Disclosure of the observer's role to others |
The extent to which participants in the setting are aware of the purpose and role of the evaluator can vary from full and open disclosure to no disclosure at all. In the biology course example, open disclosure could involve introducing the evaluator at the beginning of class to students, informing them that this person will be sitting in on some classes taking notes for a clearly stated purpose. Partial disclosure would involve identifying the observer, but not clarifying the purpose. No disclosure is rarely used, but it may occur in situations where the evaluator is trying to gain an insider perspective and there is concern that if other participants were aware of the true identity of the evaluator, they would behave differently. Obviously, there are a number of ethical issues that need to be considered before choosing to adopt the no disclosure approach. |
Duration of observations and fieldwork |
The extent and number of observations will depend on the questions and focus of your evaluation. This can range anywhere from short, single observations to long-term or multiple observations. The duration of your observations will also depend on the amount of resources you have. For a comprehensive view of how your digital library is used in the undergraduate biology class, it would be ideal to have an evaluator observe classes throughout the course, however you may not have enough time or resources for this to be possible. Some sort of schedule for sample observations would have to be worked out in advance. |
Focus of observation |
The focus of your observations is important to determine early on, although it is always subject to change. You can choose to adopt a broad focus, including almost all aspects of the setting. Or, you can choose to focus on a very narrow and specific event or behavior. Again, this depends on the specific questions of interest in your evaluation. Using the biology course example, you may choose to focus your observations on teacher use of the digital library's resources during classes, making note of things such as: (a) comments made during use, (b) any technical issues experienced during use, and (c) tasks for which digital library resources are used. It is often helpful to make an observation guide or some sort of form that lists the types of behaviors or events you are particularly interested in observing. |
Here are some other factors that should be considering when conducting observations:
Setting |
Where are you? A library? A classroom? A lab? An auditorium? Starbucks? A home? What does it feel like to be there? Is the space amenable to the use of a digital library? Are people comfortable in this space? |
Objectives |
Why are people here? Who is in charge (if anyone)? Are activities self-directed? Who maintains goal directed behavior? To what degree are people on task? What is the assumed or stated goal of using a digital library in this context? |
Implementation |
Are things going as planned? Does the digital library technology work? Who handles problems ? Are people confident in their use of the digital library? |
Interactions |
How do people interact among themselves? Is the atmosphere formal? Informal? Friendly? Unfriendly? How are you viewed? Do people share information about what they are finding with the digital library? |
Nonverbal Behavior |
What does body language tell you about this digital library? Are people interested? Going through the motions? Intimidated? Confused? Exhilarated? |
Unobtrusive Factors |
What areas of the digital library are heavily used? What areas are less used? Can people hear audio components? Are they able to print easily? |
Unexpected Things |
What did you see you didn't expect? What surprised you? What delighted you? What worried you? |
Before going somewhere to conduct field observations , it is useful to write down your expectations. Try to describe any biases or prejudices that might affect your observations. Having these expectations on record before observing provides you with a better basis for interpreting what you actually saw when you observed. Thinking back on the biology course example, you might write:
I am going to observe undergraduate students using a digital library in a large section biology course at a huge state university. I expect the course to be innovative because all students are expected to bring their laptops to the classroom. There is wireless Internet access across this entire campus. The instructors in this course have a reputation for innovative applications of technology in their teaching. In fact, these instructors developed one of the major collections for this digital library. All the students may not be “whiz kids” with respect to technology, but most have higher than average computer literacy. Their enthusiasm about biology may vary in that for many of them this is a required course. It will be interesting to see how the digital library collections are integrated into the course activities. I am a little nervous about being in an undergraduate classroom again. After all, it has been a long time. I wonder if I can relate to the students of this generation. Fortunately, there are nearly 300 students in the large lecture hall, and I should be able to blend in. |
There are several guidelines for recording your observations :
- Collect field notes. This is not optional! Do not trust everything to your memory. You'll forget things or worse invent things you did not really see.
- However, don't try to write down everything. No one can do this! Record your notes in some sort of outline or shorthand style that fits your experience and skills. Flesh out your notes with more details as soon as you can.
- Make your notes as descriptive as possible. Do not try to interpret what you are seeing at the same time you are describing it. Try to separate description from interpretation.
- Add interpretations and your own feelings later.
Separating observations from interpretations is not easy. Here is an example of poor observation notes followed by a better example:
Poor notes |
The lecture hall is a warm and friendly place. The lecturer clearly loves her students. There are lots of cool things the instructor does with the digital library resources. |
Better notes |
|
As soon as possible after you have completed your observations , you will want to go over your notes and write down everything else you can recall concerning what you have observed. It takes practice to develop good descriptive skills, and it will inevitably take some time and training to develop these skills.