User studies

Context

In an article, "What is enough? Satisficing information needs", Prabha et al. (2007) pose the question: how do users know when to stop searching for more information, when the information space is so saturated that there’s no guarantee all the relevant information has been identified? In the case of user studies, that guarantee is particularly unforthcoming.

It is one of the most researched and significant topics in library and information studies, but it also remains one of the most elusive as there is no convenient definition available to help researchers get a handle on it.

This article, then, will of necessity be an incomplete study of the topic, focusing on some of the main models that appear in the literature, as well as a selection of research studies. It will start by looking at key writers and theorists, before looking at the methods employed by user studies, and finally at some of the findings.

User studies can be anything from a survey of its users by a particular library, perhaps using LibQUAL+™, through a small-scale study of the information behaviour of a group of knitting enthusiasts (Prigoda and McKenzie, 2007), to a deep log analysis of transactional log files of Elsevier journals to find data on users (Nicholas et al., 2008).

It encompasses human-computer interaction and human information behaviour, and on the more pragmatic side, usability studies.

Few researchers provide a definition (the nearest is probably that provided by Wilson (1981), see part 2, Methods and methodology), but broadly speaking, user studies concerns the way the user behaves in the search for information to meet his or her needs.

Unsurprisingly, greater attention has been paid to more formal information requirements within a framework supplied by the library.

As far as the latter is concerned, the focus has moved from the library as a whole, and particularly the library as place, to the services provided, which, post the digital revolution, tend to be electronic and online.

Methods and methodology

User studies are subject to a very wide range of research approaches and methods, from small-scale qualitative studies to deep log analysis. I shall start with attempts by Wilson and others to develop a theoretical framework, before going on to consider the main research methods.

Theoretical framework

Beaulieu (2003) describes how Sheffield academics Wilson and Ellis helped establish a conceptual basis for user studies. Wilson’s (1981) work on user studies came out of a very long project on communication and information flow in social services departments. His model of information behaviour proved highly influential: according to Bawden (2007) it has been cited over 100 times. Whereas researchers had previously examined use, Wilson preferred to look at the actual process of information seeking, in other words the why rather than the how, and at the user in his or her environment, social, personal and professional. He provided graphics of his model as follows:

 

Figure 1. Model of information behaviour (Wilson, 1981)

 

Figure 2. Model of information seeking (Wilson, 1981)

 

Ellis derived his model (1989, quoted in Beaulieu, 2003) from his own empirical work (later verifying it with further studies), and explored patterns of information seeking such as starting, chaining (following citations), browsing, differentiating (filtering out information one did not need), extracting, verifying, and ending. Wilson subsequently revised his own theory to incorporate Ellis’ research.

The last 2006 issue of the Journal of Documentation was devoted to a revisiting of significant research papers 25 years on, including Wilson’s paper. In the same issue, Wilson commented ironically that although 25 years ago he had pleaded for more qualitative studies, this was now the dominant paradigm in this area, leading to a plethora of small-scale studies. What was now needed, he argued, was for large-scale research programmes lasting several years.

"I do not believe that qualitative methods should be the sole means of investigating information phenomena, and I regret the extent to which researchers today carry out one qualitative study after another without attempting to put the ideas that emerge to the test of the Law of Large Numbers" (Wilson, 2006a).

He also pointed to the proliferation of theoretical perspectives in the area and called for a "common theoretical framework", offering his own work in activity theory. This sees human activity in terms of its cultural, social, and historical context, including the cultural rules, norms of behaviour, motivation, etc. which determine activity (Wilson, 2006b).

Prabha et al. (2007) provide a useful account of some of the models of information behaviour (although they do not cite Wilson’s 1981 paper), including that of Kuhlthau on information seeking as a sequence of intellectual stages – becoming aware of one’s lack of knowledge, identifying a problem area, defining the problem, collecting relevant information, and presenting what has been learnt. (Kuhlthau’s work is also discussed in the information management viewpoint, Niels Jørgen Blåbjerg and the Learning Objects Web development project).

Prabha et al. use role and rational choice theories to contextualize their own work. Role theory looks at the various roles people play arising from their social positions. Rational choice theory has strong applications to economics and proposes that individual actions are the basic unit of social life; individuals make a cost-benefit analysis before selecting a particular course of action.

Because one will never know the full range of choices, the concept of "satisficing" describes the process of making a judgement that one has enough information to satisfy a need. This process also involves a cost-benefit analysis of the effort involved in looking for more information.

Steinerová and Šušol (2005) put forward a model which looks at the how, rather than the why, of information seeking, based on their major study of the information behaviour of users of academic and research libraries. Their resulting data led them to propose the following types of user:

1. User Type S: Strategic and pragmatic, views information horizontally, preferring a broad range of information sources. They prefer well established, easily accessible sources, which suit their minimum effort style. They also like to be guided by mediators (such as librarians) and information systems, and to use formal judgement criteria on relevance. Surfers and bouncers, they move quickly from one type of electronic resource to another, and tend to suffer from stress over the uncertainty of searching. More likely to be female, and students.
2. User Type A: Analytical, has a deeper and more open approach to information seeking, with natural curiosity. They use multiple methods of access and search across different types of resources, using both formal and informal sources. They are less likely to find information searching stressful or begrudge the time it takes. They will also make links with their own knowledge. Like to work independently. More likely to be male, researchers and academics.

There is also a Type P, which is a mixture of the two.

Quantitative methods: surveys and experiments

The early empirical studies undertaken by Sheffield in the 1970s (Beaulieu, 2003) used a combination of methods including: questionnaires; interviews; observation; and analysis of secondary sources, such as reference enquiries. Questionnaires continue to be used alongside other methods in large-scale studies, such as the one quoted above (Steinerová and Šušol, 2005) in which the authors surveyed 16 academic and research libraries in Slovakia and 793 subjects, with a 79.3 per cent response rate. The five-part questionnaire used both Likert-scale measuring responses and open-ended questions. Interestingly they comment that they see quantitative methods only as a starting-point, and that qualitative methods should be used for more in-depth findings.

Surveys are routinely undertaken in universities in order to target resources effectively and justify expenditure, and libraries are no exception. Often a standard instrument is used, such as LibQUAL+™ or the SCONUL satisfaction survey, with a view to measuring satisfaction with services. LibQUAL+™ was initiated in the USA, but is also used in Canada, the UK and Europe. It contains a number of standard questions on service quality, grouped under headings such as "Affect of service", "Information control", and "Library as place", and it is possible for a particular library to add its own questions. The SCONUL survey contains a number of sections designed to capture demographics, use of services, and overall satisfaction with service levels. Creaser (2006) provides a useful comparison of these survey instruments, and Kyrillidou and Persson (2006) describe a survey which took place at Lund University as part of a wider application of LibQUAL+™ to 5 Swedish institutions: 2 universities and 3 hospital libraries, and which prompted 372 responses.

Details of these instruments can be found at www.sconul.ac.uk/groups/performance_improvement/surveys/, and LibQUAL+™ can also be found at www.libqual.com/.

Usability, the extent to which a product can meet users’ goals effectively, is a popular area of user studies, and one where surveys are often combined with other methods. Kani-Zabihi et al. (2005) list these methods as:

• diary studies,
• questionnaires,
• observation,
• usability testing,
• focus groups,
• interviews,
• and transaction logs.

They carried out an experiment with 48 participants completing tasks and then filling in online questionnaires. In their study of the usability of the International Bibliography of the Social Sciences database, Lockyer et al. (2006) also got small groups of students to complete tasks, and then used an online survey (chosen for its ability to reach large numbers, it returned 957 responses of which 833 were analysed) for triangulation.

Qualitative methods

In their Editorial to the "Human information behaviour" special issue of Journal of Documentation, Spink and Foster (2007) comment:

"The methodologies used in the studies presented here demonstrate in particular that sample size and choice of data collection method need not be considered limiting factors to the study of complex information contexts. Indeed we can learn as much, or more, from the observation of individuals, of participation in small group, or from theoretical debates as from large-scale scientific approaches."

It is somewhat ironic that this issue, with its preponderance of qualitative methods, immediately follows that in which Wilson (2006a) calls for more large-scale studies. However, it does bear out the trend that recent user studies have tended to be small scale, in depth, of a particular group of users, using qualitative methods. Typically the latter, while they do not offer the breadth Wilson considers necessary, do offer depth and the ability to probe below the surface.

Stein et al. (2006) used focus groups for their investigation to support their hypothesis that graduate students are "in the most pain" and have difficulty with research methods. To gain an overview, and help them frame questions, they posted a survey on the Web. Gelfand (2005) used a variant of the focus group in the form of classroom discussion.

Semi-structured interviews are also a popular technique, used with various groups including partially sighted people (Beverley et al., 2007), a public library knitting group (Prigoda and McKenzie, 2007), and graduate students and librarians on how the former use the "affordances" the library offers (Sadler and Given, 2007).

You learn a lot from seeing how people do things, and it is therefore not surprising that observations are popular. Their value in usability studies is immediate as you can see users interacting with the product. A couple of examples were quoted above; another, smaller-scale usability study was conducted on the University of Minnesota library’s searching facilities (Sadeh, 2008). The small sample (two lots of eight) gave the opportunity for a much more free-flowing and spontaneous observation, when users were encouraged to verbalize their thought processes during a set of predetermined exercises. Observations of people at work have a self-evident value – for example that by McKnight (2007) of the information-seeking behaviour of a group of critical care nurses over a total period of 50 hours.

Deep log analysis

Totally different in character from qualitative studies, and presumably going some way towards Wilson’s plea for more large-scale studies, is deep log analysis (DLA). Log data can – and has – been analysed to reveal details of use, but deep log analysis is so called because it links analysis with the data of transactional server logs with another dataset containing demographic information, thus also providing information about the user. Nicholas et al. (2008) conducted a study of users of the ScienceDirect database, a comprehensive collection of scientific, technical and medical journals, books and bibliographic information. Data derived from the logs were considered alongside data from an online questionnaire which went out in May 2005 to 49,266 scholars, to which other (non-Elsevier) authors were added to create a representative population. IP addresses were then matched between respondents and log users. Log data showed, broadly speaking, who was viewing what, and when. There were a total of 757 matches between data logs and completed questionnaires, from a range of disciplines, the most popular being the life sciences followed by the social sciences, and a wide geographical spread (29 per cent of respondents each were from the USA and Western Europe).

The profiles yielded three different types of data as follows:

1. ScienceDirect log data recording searching, browsing and downloading actions:

• Type of article item viewed (abstract, PDF, etc.)
• Publication status
• Publication year
• Number of unique journals viewed in a session
• Subject of journals used
• Number of visits.

2. ScienceDirect log data recording search and navigation behaviour:

• Number of searches in a session
• Number of returned hits.

3. Questionnaire data

• Subject background and other demographic data
• Attitude towards various key statements.

The results revealed a rich portrait of electronic information behaviour by subject area (e.g. over a quarter of the searches by mathematicians were to older, pre-1998 articles). The authors term this an exploratory study, but already the results have shown similarities with other findings.

Portrait of the user

Up to this point we have looked at the methodologies, instruments and conceptual frameworks in user studies. But what picture emerges of the user? The following are generalizations across a number of research findings.

Searching behaviour

The traditional role of libraries has always been to help in the information search. The main change here has been people’s attitude towards libraries as a first port of call: rather than the OPAC catalogue or the library shelves, users generally prefer the Web and search engines, and developing their own search strategies (Gelfand, 2005; Stein et al, 2006; Steinerová and Šušol, 2005), simply because the former do not provide the user friendly search experience (Sadeh, 2008) of Google, where you can type a word or phrase into a box and be directed to different types of information source, book, journal article, etc. (Sadeh, 2008; Stein et al., 2006). People don’t want to have to spend too much time searching; they are not interested in learning complex search strategies (Steinerová and Šušol, 2005). Many libraries require the user to search across a number of different databases, when what is wanted is an overarching, easy to use interface with a unified point of entry, like that provided by North Carolina State University (NCSU):

 

Figure 3. North Carolina State University catalogue search interface

 

Sadeh (2008) describes the Primo system, which is an end-user interface sitting on top of the library system, as an intermediary between the latter and the user, with the added functionality of enabling the user to save queries, set alerts, store and tag records, and set preferences.

Kani-Zabihi et al. (2005) conducted an investigation into users’ perceptions of digital libraries and found that what the user wanted, above all, was to be able to find reliable information easily and quickly.

One common search strategy is to go direct to a particular author known to be an expert in the subject, sometimes getting papers from the author’s own website, or even contacting the author directly. Peers, for example graduate groups, are another source (Stein et al., 2006).

Control of information

Users like to have information at their fingertips 24/7. One study reported that users wanted to have their scholarly discovery interface immediately available, rather like the ubiquitous Google toolbar (Sadeh, 2008). According to Kyrillidou and Persson (2006), self-sufficiency in user of resources and services is a particular characteristic of the Swedish library users they studied, to a greater degree than British and US ones: the most desired item on the survey was "Making electronic resources accessible from my home or office".

Use of resources away from the library is likely to be particularly important to certain occupational groups, for example critical care nurses are unable to access scholarly information sources, and their needs may be for ready reference material which they can access without leaving the ward (McKnight, 2007).

How do users know when they have completed their search? The concept of "satisficing" is used by Prabha et al. (2007) in discussion of how users find they have enough information. Several studies report that they are often overwhelmed by the sheer quantity of resources (Gelfand, 2005; Stein et al., 2006). Prabha et al.’s study (2007) concludes that attitudes towards sufficiency differ according to type of user and the type of constraints they are under. Undergraduate and graduate students tend to find sufficient sources to fulfil assignment criteria and course requirements; faculty are influenced by time constraints; administrators seek enough information to be able to complete the task; some professionals, such as lawyers and engineers, place a premium on timeliness of information. Mansourian and Ford (2007) also write about their research on how to deal with people’s very real fear that they are missing important information on the Web.

Library services

Users may generally prefer the simpler, Google experience, but they appreciate what the library can offer, in particular the quality and reliability of their resources, the metadata which helps accurate searching, the tailored services (Sadeh, 2008). They just want access to all this with the simplicity of the Google interface.

The depth and range of resources is also appreciated, and criticized when it does not appear extensive enough (Creaser, 2006). The problem is, services are perceived as being librarian rather than user centric: resources are arranged by type of material and often located on different campuses (Sadeh, 2008). Just as users want virtual resources delivered to their desktops, so too do they want easy access to print resources, delivered to the campus library of their choice (Gelfand, 2005).

For some users, having delivery of materials integrated with their lifestyle is so important that the library as place, or the services provided by librarians, are not really important – these are the findings of the Swedish study, although students are an exception (Kyrillidou and Persson, 2006).

For many users, however, the library as place is still important. One study, based on a librarian meeting an undergraduate writing class, found that students did indeed appreciate being able to go somewhere quiet to study, where they could check their e-mail and use the photocopying and printing facilities.

"The library is the only centralized location where new and emerging information technologies can be combined with traditional knowledge resources in a user-focused, service-rich environment that supports today’s social and educational patterns of learning, teaching and research" (from a report, Library as Place: Rethinking Roles, Rethinking Space, published by the Council on Library and Information Resources, 2005, and quoted by Gelfand, 2005.)

However, the same study reported the effect of the environment on the users: their library was felt to look like a prison, whereas they wanted it to be more inviting, more like Barnes and Noble. They suggested that the library would be more attractive (and quiet) with carpeting, and that plants, posters, stretch out furniture, and a café would also improve matters.

Finally, library instruction is valued, but it must be ongoing, and linked to particular courses (Gelfand, 2005).

Conclusion

This article has looked at a number of user studies, focusing on theoretical models, methods and findings. The many small-scale qualitative studies, which Wilson wished to see supplemented by larger studies, have been joined by deep log analysis data. Yet while the latter provide a detailed picture of how particular disciplines differ in their use of electronic resources, many of the smaller-scale studies also offer interesting vignettes of users’ attitudes on all sorts of issues from searching to library space. Let’s hope that both large-scale and small-scale studies will continue, and complement one another to provide a rich and detailed portrait of the twenty-first century library user.

References

Bawden, D. (2007), "Users, user studies and human information behaviour: A three-decade perspective on Tom Wilson's 'On user studies and information needs'", Journal of Documentation, Vol. 62 No. 6, pp. 671-679.

Beaulieu, M. (2003), "Approaches to user-based studies in information seeking and retrieval: a Sheffield perspective", Journal of Information Science, Vol. 29 No. 4, pp. 239-248.

Beverley, C.A., Bath, P.A. and Barber, R. (2007), "Can two established information models explain the information behaviour of visually impaired people seeking health and social care information?", Journal of Documentation, Vol. 63 No. 1, pp. 9-32.

Creaser, C. (2006), "One size does not fit all: user surveys in academic libraries", Performance Measurement and Metrics,
Vol. 7 No. 3, pp. 153-162.

Gelfand, J. (2005), " Library as changing place: viewpoints from university undergraduates", Library Hi Tech News, Vol. 22 No. 4,
pp. 10-12.

Kani-Zabihi, E., Ghinea, G. and Chen, S. (2005), "Digital libraries: what do users want?", Online Information Review, Vol. 30 No. 4, pp. 395-412.

Kyrillidou, M. and Persson, A.-C. (2006), "The new library user in Sweden: A LibQUAL+™ study at Lund University", Performance Measurement and Metrics, Vol. 7 No. 1, pp. 45-53.

Lockyer, S., Creaser, C. and Davies, J.E. (2006), "Assessing the IBSS database from a novice user perspective", The Electronic Library, Vol. 24 No. 5, pp. 590-607.

McKnight, M. (2007), "A grounded theory model of on-duty critical care nurses' information behavior: The patient-chart cycle of informative interactions", Journal of Documentation, Vol. 63 No. 1, pp. 57-73.

Mansourian, Y. and Ford, N. (2007), "Search persistence and failure on the web: a 'bounded rationality' and 'satisficing' analysis", Journal of Documentation, Vol. 63 No. 5, pp. 680-701.

Nicholas, D., Huntingdon, P. and Jamali, H. (2008), "User diversity: as demonstrated by deep log analysis", The Electronic Library, Vol. 26: No. 1, pp. 21-38.

Prabha, C., Silipigni Connaway, L., Olszewski, L. and Jenkins, L. (2007), "What is enough? Satisficing information needs", Journal of Documentation, Vol. 63 No. 1, pp. 74-89.

Prigoda, E. and McKenzie, P. (2007), "Purls of wisdom: A collectivist study of human information behaviour in a public library knitting group", Journal of Documentation, Vol. 6 No. 1, pp. 90-114.

Sadeh, T. (2008), "User experience in the library: a case study", New Library World, Vol. 109 Nos. 1/2, pp. 7-24.

Sadler, E. and Given, L.M. (2007), "Affordance theory: a framework for graduate students' information behavior", Journal of Documentation, Vol. 63 No. 1, pp. 115-141.

Spink, A. and Foster, A. (2007), Editorial, Journal of Documentation, Vol. 63 No. 1.

Stein, J., Bright, A., George, C., Hurlbert, T., Linke, E. and St Clair, G. (2006), "In their own words: A preliminary report on the value of the Internet and the library in graduate student research", Performance Measurement and Metrics, Vol. 7 No. 2, pp. 107-115.

Steinerová, J. and Šušol, J. (2005), "Library users in human information behaviour", Online Information Review, Vol. 29 No. 2,
pp. 139-156.

Wilson, T.D. (1981), "On user studies and information needs", Journal of Documentation, Vol. 37 No. 1, pp. 3-5.

Wilson, T.D. (2006a), "Revisiting user studies and information needs", Journal of Documentation, Vol. 62 No. 6, pp. 680-684.

Wilson, T.D. (2006b), "A re-examination of information seeking behaviour in the context of activity theory", Information Research,
Vol. 11 No. 4, paper 260, available at: http://InformationR.net/ir/11-4/paper260.html [downloaded April 24 2008].