How do you do a service evaluation?

So how would you actually go about conducting a multi-attribute, stated-preference survey? Well, let's explore a simple example of what it might entail. Suppose you are considering providing additional services to support your digital library. For example, you wish to decide on the viability of having a reference librarian available to provide “live” help for users. There are a number of factors that would be associated with providing this additional service to your library, and a good way to gather information about how this service would be viewed by typical users is to use a multi-attribute stated-preference survey. Here are the steps involved in this approach:

Step 1: Identify the different service attributes for which you want user input.

For our example some attributes might include:

• Number of hours reference librarian is available for live help
• Method of communication between librarian and patron (i.e., online chat, email, or telephone?)
• Price users are willing to pay to have a live reference librarian

Step 2: Identify levels of attributes that you want to explore.

One good way to do this is to create a chart:

Step 3: Design the survey

Surveys typically begin with easy questions, to help focus the participant on the subject at hand, before moving into the more “thought provoking” questions. (Guidelines for constructing surveys can be found in Chapter 8 of this Guide.) You may want to begin your survey with questions that gather demographic information (e.g., age, position, etc.) or information about the respondent's knowledge of or use of the digital library. In the next section of the survey, you can begin presenting your choice experiment. You'll want to be sure that the options you list in your questions are ones that would be within the range of possibilities. You certainly don't wish to create expectations for a new service that is simply not feasible.

Here is an example of what a question might look like using our reference librarian example:

Example question:

Which of the following systems do you prefer?

1. Existing system, no reference librarian, no extra costs
2. Reference librarian available 8 hours/day; available via email; pay fixed price per use
3. Reference librarian available 8 hours/day; available via online chat; pay fixed price per use
4. Reference librarian available 8 hours/day; available via telephone; pay fixed price per use

This question specifically explores what type of contact (if any) users would prefer to have with a reference librarian given that it would only be available for 8 hours/day and would have a fixed price. A good multi-attribute, stated-preference survey would explore all of the possible combinations of hours available (4 levels), methods of communication (3 levels) and price for use (5 levels). Because there are so many available combinations (4x3x5 = 60), you would probably not want to present all 60 choices to every user, because by the end of the survey all the choices would begin to look the same! But by assigning a set number of questions (perhaps 20) for each user to provide feedback, you could get the information you want, provided the options are logically presented across the range of different surveys.

Step 4: Administer survey, analyze results, and present findings.

The final steps in conducting a multi-attribute, stated-preference survey would be to administer it to the relevant group of people, analyze the results, and present your findings to the decision makers in a timely manner. Today, many surveys are administered over the WWW. This seems particularly appropriate if you are surveying typical users of a digital library. However, if you are seeking input from non-users, having the survey available via the Internet alone may introduce an unacceptable level of bias because people who are not online won't be able to access your survey. You'll need to consider alternative methods of collecting the surveys such as regular mail, telephone, or person-to-person. Recommendations about analyzing and presenting your findings can be found in Chapter 10.

When attempting to influence decisions about the types of services to be offered in a library, digital or otherwise, it only makes sense to consult with the users. Using the multi-attribute, stated-preference technique is an insightful way to gather information from actual users of the digital library about the services they value and the ones they would like to have.

Other common techniques for evaluating the services of a digital library include the use of interviews, focus groups, and observations to determine patron satisfaction with various library services. One important thing to note when doing user evaluation of services is to take into consideration the difference between use value and option value (Choudhury et al., 2002). Use value refers to the value attributed to a service or feature by actual users of the service. Option value is the value attributed to a service or feature by individuals who might use the service in the future but do not currently use it. Often individuals may place a high value on a service even if they currently do not use it.

If influencing decisions related to services provided by the digital library is a goal, then focusing your evaluation on service performance based on user feedback is the way to go. Establishing the degree of user satisfaction with existing services and revealing user desires for alternative services through evaluative methods should guide future decisions concerning the services your digital library will provide. Of course, you don't want to ask your users to respond to surveys too frequently or they'll ignore them. Only do a service evaluation when important decisions need to be informed.