What is information retrieval evaluation?

Information retrieval evaluation is twofold, user-oriented and systems-oriented. From the user perspective, information retrieval evaluation is appropriately focused on evaluating how effectively and efficiently a user's search for information meets his or her needs or interests. From the systems perspective, information retrieval evaluation is focused on evaluating the effectiveness and efficiency of the retrieval system that is at the core of any digital library.

If you have to focus on one form of information retrieval evaluation over another, the user perspective is recommended. In user-orientated evaluation, emphasis is not on the user's ability to conduct “good” searches, but rather on the user's experiences with the information retrieval tools presented by a digital library. No matter how comprehensive a digital library is in terms of the quality of its collections nor how sophisticated its underlying technology may be, a digital library is of little value if users cannot find needed information in an effective and efficient manner.

Information retrieval evaluation is also not about evaluating the technical functionality of the information retrieval system, e.g., MARC-based catalogs versus automatically generated indexes. There are numerous information retrieval systems being employed in digital libraries and a great deal of research and development is focused on issues such as the utility of metadata. While these issues are incredibly important, they are beyond the focus on information retrieval evaluation as described in this Guide. What is important for our evaluation purposes is how well the information retrieval system is performing with respect to the needs, interests, and expectations of our users.

As noted above, one of the key challenges for digital libraries is the storage, organization, and retrieval of its contents. Most digital libraries aspire to have an information retrieval system that allows users to locate items of interest in the most efficient and cost-efficient ways possible. Evaluating the information retrieval capabilities of your digital library can provide useful information for making future decisions about the theoretical and technical components of your digital library in ways that maximize the effectiveness of user searches.