AI or Human? An Analysis of University Students’ Awareness of Reference Services Agent

Considering the difficulties faced by libraries in AI implementation and the research gap in users’ social-emotional aspects of AI applications in libraries, this study evaluated university students’ awareness of AI-generated versus human-created feedback (a simplified Turing test) in the scenario of reference services in university libraries. A user test was carried out on 146 Chinese university students with 5 tasks from different subject areas. Results showed students’ limited ability to distinguish AI and human agents with an accuracy rate of 49.59%. Many of them mistook human agents for AI (33.7%). The study analyzed factors affecting the judgment from three dimensions: the information about the task, AI technology, and the user. We found that the more complex the task, the more likely the students were to judge the feedback as AI-generated. When students felt the agent was knowledgeable and able to solve the task, they were more likely to judge the agent as an AI. Students also felt the AI-generated explanations were more helpful compared to the human-created ones. Students with lower AI literacy tended to judge the AI agent as a human. The study benefits AI implementation in libraries by confirming the ability of AI to provide expert services, raising the alarm bell for libraries to be replaced by AI, and calling for improvements in AI literacy cultivation.