Enhancing Higher Education Tool Evaluation through Partnerships and Automation

Higher education exists in a landscape of rapidly changing skills, knowledge and technologies and the digital learning environment to support this is growing constantly. The integration of AI in almost all tools require a new lens and a new vocabulary for us to evaluate and implement tools within this environment. Current processes involve duplication, lack of transparency and administrative inefficiencies that neglect the new environment we’re in. Our strategy involves data collection and analysis to understand tool interactions within our digital learning environment. Through University partnerships and automation, we’ve developed a framework that simplifies data collection, aligns with the digital learning strategy, and reduces administrative overhead, enhancing productivity and the teaching and learning experience.

The evaluation process consists of six stages: data collection, needs assessment and alignment, pilot and testing, evaluation and recommendations, implementation, and ongoing monitoring and support. Initially, data is collected to understand the current state, identify gaps, reduce duplication and information redundancy. The needs assessment ensures alignment with educational and functional needs and institutional policies through feature analysis, market scan, cost-effectiveness, scalability, data security and vendor support. During pilot and testing, objectives and success measures are refined, and collaboration with partners are facilitated. Key metrics include SUS, CSAT, pedagogical purpose, and tool uptake. Recommendations are made through mixed-methods analysis and benchmarking against baseline data and stakeholder endorsement. Implementation involves a collaborative detailed implementation and stakeholder communication plan with detailed support. Finally, ongoing monitoring and support ensure continuous feedback and success metrics are met, with annual reviews, feedback loops and evaluation for license renewal. These stages are distinct and not reliant on each other, allowing flexibility in the current environment with AI integration of tools, and enabling the process to start at the ongoing monitoring and support stage where necessary.  

Our presentation will discuss the process we have developed, the improvement made, the application of automation, and the impact on assessing tools in the digital learning environment to ensure transparency, prevent unnecessary renewals, and eliminate duplication across the university.