Libraries Unlimited - A Member of the Greenwood Publishing Group

Companion Website to the
Third Edition (2001) and Fourth Edition (forthcoming) of
Reference and Information Services
by Richard E. Bopp and Linda C. Smith



Excerpts from CHAPTER 10: EVALUATION OF REFERENCE SERVICES
Chapter Author: Jo Bell Whitlatch



What to Evaluate

Reference services are defined as personal assistance given by library staff to users seeking information. Reference services can be delivered in person, or virtually from a remote location, using telephone, email, or live chat. Thus, evaluating reference is fundamentally about assessing the effectiveness of the process of providing reference service to users. Many elements influence the success of reference service. Figure 10.1 illustrates the major factors that influence the nature of the reference service. All of these factors should be studied in order to obtain a complete understanding of reference service effectiveness.

Because reference service is so complex, evaluators can become overwhelmed with choices, fail to focus, and fall into the common trap of designing a study with data collection requirements that are too labor intensive. The Reference and User Services Association (RUSA) Guidelines for Information Services (http://www.ala.org/ala/rusa/protools/referenceguide/guidelinesinformation.cfm) includes a section on evaluation, which recommends emphasizing the factors that are most important to the local community, including response time, accessibility of services, value and effectiveness of services for various groups, and effectiveness in anticipating the community's needs.

The primary focus of this chapter is on evaluating reference services that libraries provide to users. Thus, the perspective these studies take is that of the user within the context of existing library reference services. However, in discussing what to evaluate, it is important to look beyond present library services and to consider conducting evaluation studies that focus upon the information-seeking patterns and behaviors of people in the library community. Evaluation of existing library services tends to address the question of how good the existing reference services are, while studies with an information-seeking perspective are better at addressing questions relating to whether libraries are offering the right type of reference services to the most important community constituencies. The information-seeking perspective is particularly important in attracting special populations of users that the library wishes to serve, such as children, youth, the elderly, people with disabilities, and ethnic populations in the community (see Chapter 12 for a discussion of developing services to meet the needs of these specific populations).

Evaluation does require staff time. The time and money invested in evaluation should have definite benefits. All evaluation should be designed to improve services. If evaluation results do not have the potential to enhance services, the resources may be better used to meet other library needs. However, failure to evaluate reference services means that services are unlikely to realize their potential for success. Continuing improvement of services does require not only planning and implementing services but also obtaining systematic information on successes and failures through evaluation. Without systematic and ongoing evaluation, library staff will never have the opportunity to obtain systematic information so essential for continuing growth and development of the reference service.

Evaluation can focus primarily on inputs to the reference process, such as assessing the quality of the reference collections, or the quality of building signage, layout, and organization of the reference desk, or ease of use of the Web site to locate reference assistance. Or, evaluation can emphasize the process of the interaction between the librarian and the user. For example, this could include the librarian and/or user behaviors associated with success in locating information. Finally, evaluation may focus primarily on the outputs or outcomes of the transaction. Was the user satisfied with the process and/or the information received? What value did the information add in the context of the life of the user?

Over time, the inputs, process, and outputs should be assessed. One very good diagnostic instrument for getting started is the Wisconsin Ohio Reference Evaluation Program (WOREP) [http://worep.library.kent.edu]. For traditional reference desk services and phone queries this instrument collects information from both the user and the librarian on sources used, process elements, such as librarian courtesy, and outputs, such as user satisfaction with the information provided. Results can be compared with similar institutions and used to set desirable performance standards. Staff with little experience in evaluation can gain good experience in planning and administering this reliable and valid instrument. Results can also suggest the most profitable areas for additional in-depth study.

Boxes (for Class Discussions)

Box 10.1 Evaluating Reference Transactions

The university reference department has received a number of comments from faculty in History, English, Biology, and Chemistry that students have not been receiving the help that they needed with their assignments at the reference desk. The reference department head has many responsibilities for collection management and instructional programs in addition to reference services, so she does not work at the reference desk and has had little opportunity to observe the reference librarians interacting with students. Because of the complaints, the dean of the library has told her that she must design an evaluation of reference desk service effectiveness.

  • What methods will provide her with the best evaluative information to help diagnose reasons for the faculty dissatisfaction with the service and possible solutions — unobtrusive evaluation, direct evaluation, user surveys, surveys of reference librarians, interviews of students and/or librarians, or focus groups?
  • How should she deal with issues related to the fact that the reference librarians might feel threatened and defensive when told that they must conduct an evaluation of their service because of the faculty complaints?

Box 10.2 Evaluating Reference Collections

The head of a public library reference department has observed that fewer users are consulting books in the reference collection. A survey of reference collection use in the past year shows that only one out of every ten books in the reference collection was used. Reference librarians in the department have various theories on why use is declining. The most popular explanations are: Business people have quit coming to the library because they think everything is available through Google; no one in the community is interested in using print materials anymore; users of the library have changed and the reference department is collecting the wrong types of print materials; the reference collection needs weeding and would be used more heavily if people could find the most current materials more easily.

  • Identify some methods that could be used to study why use of the reference collection is declining.
  • Discuss the strengths and weaknesses of user surveys, observation methods, and interviews in collecting data to discover why reference collection use is falling and what the public library should do to enhance use of the reference collection.

Box 10.3 Using Reference Statistics in Evaluation

The public library reference department has kept statistics on the reference and directional/information questions asked by all users for the past year. The reference staff have used the statistics to study the busy and slow times at the reference desk and to attempt to match reference desk staffing with the patterns that have emerged. However, reference staff would like to use the statistics to evaluate the quality of the services provided.

  • How could the reference librarians collect statistics that could be used to assess the quality of the services and go beyond just simply using the statistics to adjust staffing levels at the reference desk?

Box 10.4 Reference Services in a Joint Use Library

The president of the university and the mayor of the city have agreed to offer library services in a combined public and academic library building. At present, both the university and the city have different libraries, but it is clear that both library organizations are going to be in the same library building within a couple of years. The library directors need to determine how services will be organized and delivered to the various user groups in the new library. One of the most controversial areas is offering combined reference services. City reference librarians believe that an important part of the service is to actually provide the information that the user wants, not just tell the user how to find it. On the other hand, university reference librarians are very focused on information literacy and believe that it is important that students learn how to find information on their own once the reference librarian has provided some general instruction.

  • What types of studies could these two libraries conduct to determine whether they ought to offer combined reference services for both the university and the city communities?
  • What evaluation methods might be most helpful in determining whether combined reference services will provide the most effective services to both community members and students in the new city/university library?

Box 10.5 Evaluating Services for Specific Populations

A proud public library tradition in this Eastern community has been the role of the library in assisting immigrant populations in achieving success in their new country. Waves of immigrants from Germany, Norway, England, Ireland, Scotland, and Italy have valued the library as a place of independent learning that supplements their formal educational opportunities. In the past decade new groups of immigrants have started to settle in the area, including many Hmong from Southeast Asia and Latinos from several Central and South American countries. The reference staff have noted that few of the new immigrant populations are using the library. The few people from these new immigrant populations who are using the library are definitely not using reference services. A major focus of the library is devoted to serving immigrant populations, and the new public library director wants to increase the use of the library by this group. He has turned to the reference staff to develop an action plan for outreach to this population.

  • What should the reference staff recommend in order to develop an effective plan of action that is likely to be successful in getting more of the new immigrants into the library and using the full range of library services?

Box 10.6 Evaluating the Effectiveness of Reference Training

The city has just completed a major study of all public services to citizens in the community using "secret shoppers." Results of the study indicate that on the whole library services are good, but that the secret shoppers who asked for government information (city, county, state, or federal) received inaccurate or incomplete information. On the basis of these results the public library commits to a major training program for locating government information in response to typical user queries. Each reference staff member is required to attend more than two days of training workshops on government information. Staff members evaluate the workshops very positively. Another secret shopper study is conducted a couple of months after all staff have completed the training workshops. Results of the second secret shopper survey indicate that there has been no improvement in government information services provided by the library. Reference staff members are not certain that the results of the secret shopper survey were reliable.

  • What might the staff do to determine whether the secret shopper results are really representative of the service provided?
  • What methods might be most productive in determining why government-information reference services still do not rank highly on the secret shopper survey?

ADDITIONAL READINGS

  • Carter, David S., and Joseph Janes. "Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study." Library Trends 49 (Fall 2000): 251-65.
    Logs of over 3000 questions were analyzed in early 1999 to determine, among other things, how questions were handled, answered, and rejected. This study is an excellent example of how librarians can use question sets to learn about the quality and effectiveness of digital reference service.
  • Coffman, Steve, and Linda Arret. "To Chat or Not to Chat." Part I. Searcher 12 (July/August 2004); Part II. Searcher 12 (September 2004). Available at: http://www.infotoday.com/searcher/jul04/arret_coffman.shtml (part 1) and http://www.infotoday.com/searcher/sep04/arret_coffman.shtml (part 2).
    In response to the widespread adoption of virtual reference, Coffman and Arret question the costs and encourage librarians to explore alternatives, such as answering the phone. The purpose of this very thoughtful article is to look carefully at reference service as a whole and the methods that provide the best service. The costs and benefits of any new approach need to be carefully considered.
  • Hirko, Buff, and Mary Bucher Ross. Virtual Reference Training: The Complete Guide to Providing Anytime, Anywhere Answers. Chicago: American Library Association, 2004. 160p.  This book describes the training program for Washington's Statewide Virtual Reference Project. This model training program emphasizes active learning through a curriculum that creates a learning environment allowing library staff to actively explore, discover, and reflect on the skills needed for reference services in an online environment. The book and the online Virtual Reference Training Program (http://66.212.65.207) provide a section (chapter 9 in the book) and an online module (http://66.212.65.207/fourth.htm) on evaluating and improving reference services.
  • Janes, Joseph. "Survey Construction." Library Hi Tech 17 (1999): 321-25.
    This column is one of a series on research methods that has appeared in Library Hi Tech. This concise article covers the basics on writing good questions and designing good questionnaires.
  • Kawakami, Alice, and Pauline Swartz. "Digital Reference: Training and Assessment for Service Improvement." Reference Services Review 31 (2003): 227-36.
    This study assesses digital reference competencies in order to determine gaps in training. Observation was used both as the primary assessment method and to improve the training program.
  • Kuhlthau, Carol Collier. "Learning in Digital Libraries: An Information Search Process Approach." Library Trends 45 (Spring 1997): 708-25.
    Kuhlthau briefly summarizes her model of the Information Search Process (ISP) and the challenges for librarians to create environments for learning within libraries. She notes that the goal of the learner, which is to move from uncertainty to understanding, cannot be met by merely locating relevant, or even useful, information. To achieve success librarians must adopt strategies that are appropriate to the user's stage in the information search process.
  • Kuruppu, Pali U. "Evaluation of Reference Services-A Review." The Journal of Academic Librarianship 33 (May 2007): 368-81.
    Kuruppu provides an extensive review of the literature on reference service evaluation, with emphasis on evaluation methodologies. Sections include quantitative evaluation methods (reference statistics, surveys and questionnaires) and qualitative evaluation methods (observations, interviews, focus groups, case studies). Validity and reliability of quantitative and qualitative research results are also discussed.
  • LibQUAL: Charting Library Service Quality. Available at: http://www.libqual.org/
    LibQUAL is very useful in assessing user perspectives of library service quality by focusing on the gap between customer expectations and perceptions across various service constructs. One of the constructs of service quality is "affect of service", which includes empathy, accessibility, and personal competence (such as willingness to help users). This instrument is based on SERVQUAL, widely used for assessing service quality in private business service organizations. LibQUAL is offered by the Association of Research Libraries (ARL), but it also can be used by public and less intensive research academic libraries. This tool is useful as a general survey to identify priority areas for more in-depth study for reference service evaluation.
  • McClure, Charles R., R. David Lankes, Melissa Gross, and Beverly Choltco-Devlin. Statistics, Measures and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. Syracuse, NY: Information Institute of Syracuse, School of Information Studies, Syracuse University; Tallahassee, Fla.: School of Information Studies, Information Use Management and Policy Institute, Florida State University, 2002. 104p. Available at: http://www.webjunction.org/do/DisplayContent?id=11813.
    In this report the authors begin to identify, describe, and develop procedures for assessing digital reference service in order to improve the quality of service. The manual covers a wide variety of measures for digital reference: descriptive statistics, log analysis, user satisfaction measures, cost, staff time expended, and other options. The manual also includes useful sample forms and survey instruments.
  • Novotny, Eric, ed. "Assessing Reference and User Services in a Digital Age." The Reference Librarian 95/96 (2006): 1-230.
    This collection of articles explores the evaluation of reference services from a variety of perspectives. Approaches vary from case studies of individual libraries to discussions of the best methods and approaches.
  • The Outcomes Toolkit. Available: http://ibec.ischool.washington.edu/toolkit.php
    This resource is very useful for conducting evaluations based on outcomes (rather than inputs or process measures). The Outcomes Toolkit provides a four step process: getting started, collecting data, analyzing data, and using what you find. The Toolkit was developed by the IBEC (Information Behavior in Everyday Contexts) team, which is composed of researchers and graduate students from the University of Washington and the University of Michigan. Project leaders are Joan Durrance and Karen Fisher.
  • Pomerantz, Jeffrey, Lorri M. Mon, and Charles R. McClure. "Evaluating Remote Reference Service: A Practical Guide to Problems and Solutions." Portal: Libraries and the Academy 8 (January 2008): 15-30.
    This paper identifies key methodological issues affecting the quality of data in the evaluation of virtual reference services. The authors propose strategies for practical ways in which libraries can improve the overall quality and usefulness of data gathered in such studies.
  • Reference and User Services Association (RUSA). Research and Statistics Committee. Available: http://www.ala.org/ala/rusa/rusaourassoc/rusasections/rss/rsssection/rsscomm/
    rssresstat/researchstatistics.cfm
    This web site provides annual bibliographies of recently published research on reference and annual programs on assessing reference services, as well as information on useful web sites for reference research. RUSA also publishes a quarterly magazine, the Reference & User Services Quarterly, which frequently features articles on reference research.
  • Rimland, Emily L. "Do We Do It Good Well? A Bibliographic Essay on the Evaluation of Reference Effectiveness." The Reference Librarian 98 (2007): 41-55.
    Rimland examines recent (1994-2006) research into the evaluation of reference effectiveness. Topics covered include unobtrusive studies, obtrusive studies, emerging trends, and research needs.
  • Ross, Catherine Sheldrick, and Patricia Dewdney. "Negative Closure." Reference & User Services Quarterly 38 (1998): 151-63.
    This study uses observational techniques to explore librarian behaviors related to terminating the reference transaction without providing a helpful answer, including unmonitored referral, implying that the user should have done something before asking for reference help, trying to get the user to accept easily found information, claiming the information does not exist, and going off, never to return.
  • Virtual Reference Desk. Evaluation and Research. Available at: http://www.webjunction.org/do/Navigation?category=11649.
    This compilation includes resources such as white papers, quality standards, policies, and other materials focusing on assessment and its importance to virtual reference practices and procedures.
  • Von Seggern, Marilyn, and Nancy J. Young. "The Focus Group Method in Libraries: Issues Related to Process and Data Analysis." Reference Services Review 31 (2003): 272-84.
    This paper covers the basics of planning for collecting data using focus groups. Ethnograph software for coding and analyzing focus group data is discussed and a useful example of the coding is provided. An annotated bibliography includes focus group studies in a library setting.
  • Wallace, Danny P., and Connie Van Fleet, Eds. Library Evaluation: A Casebook and Can-Do Guide. Englewood, Colo.: Libraries Unlimited, 2001. 237p.
    Chapter 3 (pp. 79-115), written by Kathryn Dana Watson, covers evaluating reference and information services. The chapter includes case studies using the Wisconsin Ohio Reference Evaluation Program (WOREP) and assessing reference behaviors with unobtrusive testing.
  • Watson-Boone, Rebecca. "Academic Librarians as Practitioner-Researchers." The Journal of Academic Librarianship 26 (March 2000): 85-93.
    Watson-Boone identifies seven steps common to both research- and practice-based problem solving: identify the true problem that needs attention; define the various ways to solve the problem; select the process that appears to have the greatest chance of working; set out criteria against which to measure the success of specific problem-solving efforts; carry out the effort; evaluate what occurs; and reflect on whether the results have solved the problem to a significant degree.
  • Whitlatch, Jo Bell. Evaluating Reference Services: A Practical Guide. Chicago: American Library Association, 2000. 226p.
    This guide provides information on the basic techniques that are commonly used to collect data for evaluating reference services and also contains an extensive annotated bibliography (pp. 113-213) of reference research studies. Readers wishing to use a method or study a certain type of reference issue can use the research studies as guides to get started on their own research project.