Why evaluate
Planning
Service eval
Usability eval
Info retrieval
What is it
Technique
Case study
Refs & tools
Bibliometrics
Log analysis
Surveys
Interviews
Observations
Experiments
Reporting
References
Credits
Search
PDF version

Information retrieval case study

An evaluation conducted by Berenci, Carpineto, Giannini, and Mizzaro (2000) is a notable example of an information retrieval evaluation. Berenci et al. examined how visual displays could be used to increase the effectiveness of using ranked-output retrieval systems, which are commonly used in web-based search engines such as AltaVista. They developed a system called VIEWER (VIEwing WEb Results) which acts as an interface to any selected search engine.

VIEWER acts to provide a graphical representation of the search results along side the ranked search results provided by the search engine. The visualization of the data displays red horizontal bars that each represents the number of “hits” for each sub-query (which are formed by combining the number of query terms). Users are then able to click on any of the bars to select the associated documents. The advantage of having such a system is that users do not have to search through all retrieved documents to find the information they are looking for. Rather, the VIEWER system allows users to immediately select documents that have the relevant combination of terms.

Berenci et al. (2000) decided to do a comparative study of their VIEWER system and a typical web-based search engine (in this case AltaVista was chosen). Their goal was to evaluate the effectiveness of VIEWER in comparison with AltaVista in a realistic search situation. They hypothesized that the VIEWER system would allow users to focus on the relevant document summaries and reformulate future queries. Berenci et al. decided to use precision and the raw number of relevant documents found as their criteria for comparison. Questionnaires were also used to measure user satisfaction, utility of the system, and the usage of the views in the VIEWER system.

The evaluation results showed that although AltaVista retrieved many more documents than VIEWER, the number of relevant summaries retrieved was very similar for both systems. Thus, AltaVista retrieved many more non-relevant documents than the VIEWER system; in short, it wasn't as efficient as VIEWER. Also, the precision values when using the VIEWER system were markedly better than those obtained when using AltaVista. Thus, Berenci et al. (2000) were able to use the measures of precision and relevant document retrieval as support for adopting an information retrieval system such as VIEWER.