From Hesitation to Integration: challenges of implementing a chatbot in the library

Change and transformation have been constants in the library sector for many years.  Libraries have continually needed to innovate and create efficiencies – to do more with less.  From July-Nov 2024, the University of Queensland Library piloted the use of a 24/7, client-facing Chatbot.  In this presentation, we will share some of our learnings, both from a technical and human-centred standpoint.

A first for UQ, it was the product of a collaboration between UQ’s Information Technology Services, the UQ Library, and Microsoft (via technology partner Blackbook.ai).  The Library Chatbot is built on Microsoft Copilot Studio technology and it sits alongside the Live Chat service for clients to opt into using.  It was designed to respond to Tier 0 questions based on content sourced from specific URLs.  

Our chatbot utilises retrieval-augmented generation (RAG) to provide more accurate and relevant responses than traditional chatbots.  It does not learn, and unlike ChatGPT, it is not a conversational tool.  This introduction of a generative-AI technology has created the potential for library services to be transformed through scaling up and providing support for clients 24/7.

In readiness for the Chatbot’s introduction, 95 FTE days of User Acceptance Testing was performed by library staff from multiple teams, including front facing client services staff whose roles would be impacted for better or worse.  In response, web content was tweaked, the number of URLs it had access to was altered, and changes to Copilot itself were made by Microsoft when errors arose.  As a form of black box technology, whereby no one has full knowledge of the bot’s internal workings, our learning curve was steep.  Testing was repeated in October, prior to the launch of a large-scale web content rewrite.

Throughout the implementation we navigated challenges including potential reputational risk through the provision of wrong information to clients, end user frustration, lack of support from the academic community, missed identification of students in crisis, and threats to the university. In addition to these challenges, we were keenly aware of the impact on our staff and took steps to engage and support them throughout the journey.