Why evaluate
Planning
Service eval
Usability eval
Info retrieval
Bibliometrics
Log analysis
Surveys
What are they
Technique
Case study
Refs & tools
Interviews
Observations
Experiments
Reporting
References
Credits
Search
PDF version

How do you conduct surveys?

There are a number of issues that must be considered when one decides to carry out a survey. There are issues of sampling (Who will receive the survey? How can I motivate them to respond?); administration (How will I administer the survey? Paper? E-mail? Web-based?); design (What will my survey address? What scales do I use? What is the best wording for questions?); analysis (How will I process and synthesize the data I get?); and reporting (How can I communicate the results clearly?). All of these issues are important to consider. According to Fink (1995), the best examples of surveys have the following characteristics:

  • Specific objectives
  • Straightforward questions
  • Sound design
  • Sound choice of population or sample
  • Reliable and valid instruments
  • Appropriate analysis
  • Accurate and timely reporting of results

Specific Objectives

Defining the objectives of your survey is important in helping you determine the questions the survey should ask and the information that you will gather. As emphasized above, the overall goal of any survey should be informing decision-making. More specifically, an objective is a statement of the intended outcome of the survey. Here are two examples of survey objectives with accompanying questions:

Objective: Identify the technology skills and experience of typical digital library patrons.

Sample question: Using a 1 (not at all) to 10 (expert), rate how familiar you are with the following technologies:
__ personal computing
__video production
__digital photography
__MP3 music

Objective: Determine frequency of use of the digital library.

Sample question: How often do you use the ____ digital library?
__This is my first visit.
__Rarely (less than once a month)
__Monthly (1 to 3 times a month)
__ Weekly (1 to 3 times a week)
__Daily

Straightforward Question and Responses

Of course, for your respondents to be able to provide you with the information you are seeking, you should ask questions in as clear, precise, and straightforward a manner as possible. For the most part, questions should be focused (dealing with only one thought or issue at a time). Failing to use correct grammar and syntax will decrease the survey's credibility and dampen participation. Questions can take one of two forms: open-ended or closed. Open-ended questions require respondents to generate their own answers using their own words. Questions in which the respondent is required to select responses from a pre-determined set of answers are closed questions. Open questions are useful when you do not know the types of responses to expect to a question, and for gaining a respondent's unique perspective about an issue in his or her own words. The difficulty with open-ended questions is in the analysis of the responses, which requires training in qualitative research. Closed questions are often more practical because their results lend themselves to statistical analysis. However, closed questions are more difficult to write because the response choices must be known in advance. Here are examples of open and closed questions:

Open-ended question:

What do you value most about the digital library? __________________________________

Closed question:

How many times during the past week did you use the digital library?
__never __1 time __2-3 times __4-5 times __6 or more times

Sound design

Choosing the type of survey design you will employ depends upon the decisions you need to make and the aligned objectives of your research. You can choose to do a comparative design or a descriptive design. A comparative design involves surveying two or more groups distinguished by variables of some importance to your evaluation. For example, you might wish to compare the attitudes of professors versus students toward a digital library. Descriptive designs are employed when you are looking to gather information from whole groups, e.g., a survey of the population of your patrons concerning needed extensions to the digital library.

Sampling

Ideally, you might want to administer a survey to the entire population of your users, but this is rarely feasible or even necessary. One of the most important things is to ensure you have a representative sample. A representative sample shares all the important characteristics (such as age, gender, skill level, etc.) of the larger population that you are interested in studying. To select a representative sample there are a number of sampling techniques that can be used, but before you decide on a sampling technique you should establish the eligibility criteria for your study. The eligibility criteria are those characteristics that you deem respondents need to have in order to complete the survey. For example, if you are interested in how middle school teachers are using your digital library in their classroom, you might set the eligibility criteria to include:

  • teachers of grades 6-8 (because you are only interested in middle school teachers)
  • teachers who have used the digital library for at least 6 months (because you are only interested in experienced users of the digital library)

Establishing eligibility criteria helps you determine who among the general population is eligible to be included in your sample population. Once you have established those individuals eligible to participate, you need to choose a suitable sampling technique to find participants for your survey. There are advantages and disadvantages to using different methods for selecting a sample. Typical sample selection methods include: random, cluster, convenience, and snowball.

Random sampling:The basis of random sampling is that every individual has an equal chance of being selected. One way to generate a random sample is to apply a table of random numbers to a list of prospective participants. Or you can use a random number generator available on the Web at: http://www.random.org/. An advantage of using a random sample is that the results are relatively unbiased because of the equal probability for participants to be selected.

Cluster sampling: A cluster is a naturally occurring unit such as a school, a county, city, state, etc. With cluster sampling you can randomly select from among the clusters, and then survey all the members of the cluster or a random subset of them. Note here that the resulting sample may not be representative of the other clusters, as well as not representative of aspects not covered by the cluster.

Convenience sampling: This method relies on using an already available group of individuals. For example, surveying the professors and students in a university with which you are associated may be considered a convenience sample.

Snowball sampling: Sometimes it is difficult to find participants who meet your criteria for inclusion in your evaluation. However, most often you are able to find at least one or two participants who meet your criteria. Subsequently, these participants will likely be able to identify one or two other individuals who will also meet your criteria, and thus your sample snowballs.

Reliable and Valid Instruments

Reliable survey instruments allow you to obtain the same information each time that you use it (assuming no intervening circumstances). A reliable survey instrument is said to be relatively free of “measurement error,” which is important in ensuring that results represent individuals' “true” attitudes, opinions, etc. You can increase the reliability of your survey instrument by doing some of the following: (a) ensuring the reading level of your survey is appropriate for your population, (b) ensuring your questions are clearly written, and the directions are easily understood, and (c) ensuring you administer the survey in appropriate ways and in appropriate environments (to ensure environmental factors have a minimal impact on participant responses).

Valid survey instruments must be reliable first, but in addition they should measure what they are intended to measure. For example, if a survey's aim is to find out about the pedagogical beliefs of teachers who use your digital library, the results from your survey should be judged by content experts to measure teacher pedagogical beliefs as well as be consistent with other measures of pedagogical beliefs. Validity is often discussed along four dimensions: content, face, criterion, and construct.

Content validity is the degree to which a measure appropriately assesses what it was designed to measure (as briefly discussed above). Content validity is often established by basing survey construction upon models or conceptual frameworks found in the literature. For example, if you were interested in surveying digital library users' information-seeking behaviors, you could base your survey on the literature about how individuals typically seek out information in digital libraries.

Face validity deals with how well a measure appears to address what it was intended to address. Does it ask the important questions needed, and does it use the appropriate language to do so? Unlike content validity, face validity is not grounded in the literature. Judging face validity is subjective, but the process is enhanced when experts are used.

Criterion validity is focused on one of two things: predicting future performance (known as predictive validity such as is found in test like the Graduate Record Exam (GRE)), or comparing responses to those from more well-established surveys (known as concurrent validity).

Construct validity is the degree to which a survey is able to distinguish between participants who do and do not have certain characteristics. Construct validity can typically be established in two ways:

  1. establishing that your survey has convergent validity (correlates with measures of similar characteristics) and has discriminant validity (does not correlate with measures that are not similar), and
  2. establishing that your survey can discriminate between groups of individuals on important variables.

Appropriate analysis

Depending on the type of survey you have constructed, analysis of survey data can employ statistical or qualitative techniques. Surveys that are primarily composed of closed questions lend themselves to statistical analysis. Typical goals of statistical analyses are to produce:

  • Descriptions (e.g., the backgrounds of respondents; responses to questions)
  • Relationships (e.g., connections between variables)
  • Comparisons (e.g., between subsets of your sample population)
  • Predictions (e.g., of how individual variables such as gender or age relate to reported behaviors)

There are numerous qualitative data analysis techniques that can be used to analyze the data from open-ended questions, but a common approach is to categorize common responses. You can go through all participant responses to categorize them based on similar main ideas or issues. By doing this, you will gain a better sense of which responses were most frequently given by respondents and thus have some indication of the more important issues for your survey sample.

Accurate and timely reporting of results

You can conduct the best survey in the world, but if you fail to report the results accurately and in a timely matter, it is all for naught. It is easy to mislead people, intentionally or unintentionally, with graphs and tables that present data in a skewed manner. After 50 years, Darrell Huff's (1954) book How to Lie with Statistics remains a popular critique of how statistics and graphs are often used to misinform. It is doubtful that evaluators of digital libraries would intentionally mislead their stakeholders, but care must be taken to present the results completely, warts and all. It is equally important to report survey results in a timely manner so that the decisions that the evaluation is intended to inform haven't already been made. Rather than waiting to compile results into a long printed report that few decision makers will read, it is often more effective to report findings in brief bulletins or executive summaries.

The above characteristics of good survey research are important for all types of survey research including written surveys, interviews, and focus groups. Next, let's take a more in-depth look at how written surveys can be used in digital library evaluation research.

Questionnaires

There are a number of important issues to keep in mind when designing a written questionnaire, but one of the most important is the construction of the questions. Here are several tips, based partly on guidelines suggested by Fink (1995), to help you design a questionnaire that will be effective in obtaining the information needed to influence decision making concerning your digital library:

Keep questions short and specific. Try to avoid asking two things in one question such as “When and how often do you use the digital library?” Be as specific as possible when asking questions to ensure that respondents will give you the type of information you need. For example, if you are interested in the times of day that individuals are using your digital library asking “When do you use the digital library?” might not be the best question to ask. Asking more specific questions such as “What times of day do you most often use the digital library?” will prompt respondents to give specific times rather than other ambiguous answers. Asking vague questions only results in difficult-to-interpret or unhelpful answers. Also try to avoid using run-on sentences.

Use vague qualifiers with caution. Vague qualifiers are adverbs like “usually” that can mean different things to different people. Using such words makes it difficult to interpret participants' responses because you cannot be sure how they interpreted the qualifier (e.g., “often”).

Use jargon, abstract terms, and acronyms with caution. Unless you are able to carefully assess that your survey participants understand jargon (e.g., mirroring), abstract terms (e.g., economy of execution), or acronyms (RMB for right mouse button), you should avoid them in your questionnaire. Alternatively, you can provide definitions of terms with which you suspect your participants may be unfamiliar. You will also need to define any abstract terms that may have multiple meanings to clarify the particular meaning that you want participants to apply in completing your survey.

Organize questions from easiest to more difficult. Typically survey designers order the questions from easiest to most difficult and complex. Partly this is to ease participants into the survey and not scare them off with the first question! Although participants can answer printed questionnaires in any order they chose, this “rule” is typically still followed by most survey designers. Web-based surveys can be designed to gradually reveal questions, perhaps from simple to complex.

Organize questions in a logical order. Related questions should be grouped together, and the questionnaire should be organized in such a way that questions are asked in an order that makes logical sense. Typically asking general questions before more specific ones works well.

Have a rationale for where you place demographic questions. Demographic questions are most commonly placed at either the beginning or the end of a questionnaire. Survey researchers differ on where they believe demographic questions should be placed. Those who maintain they should be placed at the beginning of the questionnaire believe so for two reasons: (a) demographic questions are easy for respondents to answer (thus it falling line with the reasoning of starting with easier questions), and (b) respondents who turn in incomplete questionnaires tend to leave the last part, rather than the beginning unfinished. Advocates for placing demographic questions at the end of the questionnaire disagree with this logic and instead insist that: (a) since most questionnaires are accompanied by a cover letter describing the topic of the survey and encouraging participation, starting with demographic questions negates the purpose of the cover letter, (b) many people find demographic questions boring and thus may not be motivated to complete the survey if they are the first questions asked, and (c) beginning with easy questions that engage the participant in the topic of study will increase the response rate and reduce missing data. Wherever you choose to place your demographic questions, be sure that you have thought about the potential implications of placing them where you do.

Utilize closed questions whenever possible. Although you may be tempted to add open questions (e.g., What are the benefits of using our digital library?), be cautious. Open questions are easy to pose, but often time-consuming and difficult to analyze. In addition, if you have very many of them in your questionnaire, your response rate will go down because people simply do not have the time to respond.

Craft closed questions with care. Closed questions are easier to analyze, but only if they have been well-written in the first place. Perhaps the most difficult challenge is creating an exhaustive list of responses for closed questions. If you are not sure that your categories are exhaustive, consider use of an “other” answer category that allows respondents to clarify what “other” is for them. It is often difficult to create mutually exclusive answer categories, but you should strive to do so. At the same time, you must attempt to keep answer alternatives short and precise. It is also essential to inform respondents whether they can select only one answer or multiple answers.

Pilot test your questionnaire. Never distribute a questionnaire before you have pilot tested it with several different audiences. No matter how experienced you are at preparing questionnaires, there will always be things you miss. Start testing the questionnaire with colleagues, and gradually increase the testing with people who are more like the eventual targeted respondents. Expect to go through three to ten versions of your questionnaire, depending on its complexity and length.