Transactional log analysis case study

An exemplary example of using transaction log analysis in evaluating a digital library is the evaluation done by Jones, Cunningham, McNab, and Boddie (2000) from the University of Waikato in New Zealand. Jones et al. conducted an extensive log analysis of the New Zealand Digital Library, focused on the Computer Science Technical Reports Collection. Evaluating and improving upon the user retrieval interfaces was the driving force behind conducting usability studies as well as employing transaction log analysis techniques.

Jones et al. (2000) discovered a number of interesting findings from their analyses, but what is most useful to note is that they went beyond the numbers to try to look at how the information could be used to improve users' experiences using the library. For example, their log analysis revealed that users rarely changed the default settings for query or results display options. The evaluators decided that this finding could mean one of two things: (a) the default settings were appropriate for the majority of users needs, or (b) users tended to accept the default settings regardless of what they are. Jones et al. concluded that since they could not know from only the log analysis which of these two hypotheses was true, extra care should be taken in creating the default settings and ensuring they are the most efficient settings possible, as it may be likely that users will accept the default settings as they are.

To give you more ideas of the types of analyses you can perform with your log files, here are additional examples of the analyses performed during the evaluation of the New Zealand Digital Library's Computer Science Technical Reports Collection (Jones et al., 2000).

Location and affiliations of users

Log file analysis can provide information on the location of users, as well as their affiliation, e.g., educational institutions =.edu versus commercial interests =.com.

Boolean vs. ranked queries

The frequency with which users opt for using Boolean or ranked search queries can be calculated. This information can help determine an appropriate default setting.

Query complexity

The complexity of search terms used, e.g., one or two-word search terms versus five or six-word search term, can be generated from the log files to provide insight into how users approach searching in your system.

Query terms

Analysis can be done to determine the most commonly searched terms. This information can be used for structuring term indexes in the system.

Term specificity

Knowing whether users are using overly general or overly precise terms while searching your digital library collection can be helpful in understanding precision and recall measures.