Experimental methods case study

In the online Journal of Digital Information, Salamapsis and Diamantaras (2002) describe an experimental evaluation of two different search system architectures for digital libraries. The authors were seeking to determine the relative effectiveness of the open hypermedia system (OHS) for retrieving information in comparison to web browser-based searching (WWW).

Twenty-four subjects were randomly assigned to either the OHS treatment or the WWW treatment. Each subject was tested individually by being given the same information query problem. They each had thirty minutes to find as many relevant documents as possible from a predefined digital library. Recall (the proportion of relevant documents that are retrieved from the collection of all relevant documents) and precision (the proportion of documents retrieved that are relevant to the information being sought) were calculated for the search results of each participant.

Although it was found that the subjects using the OHS treatment were more effective in terms of both recall and precision, the results were not statistically significant. Salamapsis and Diamantaras (2002) presented several arguments for the importance of their findings, but in the end, they concluded that “because the results cannot be validated statistically, the views and statements reported in [their] paper should be regarded as indicative and tentative.” The “no significant differences” problem has been evident in decades of research and evaluation in educational contexts (Clark , 2001). Even if this evaluation had revealed statistically significant differences in the OHA and WWW search tools, there would be no guarantee that the results found in such a controlled experiment would generalize to the rough and tumble world of real world digital library usage.