We know effective learning hinges on feedback (Hattie & Temperley, 2007). Yet, it is time consuming, requires the right expertise and needs to be timely in order to work. At Multiverse, we have investigated how to use AI to tackle these challenges and evaluate whether AI feedback is as effective as traditional models.
Learner feedback is traditionally viewed as a passive transmission of information from a teacher to a learner. However, the modern learning landscape emphasizes a more engaging and responsive process centered around the learner, where the exchange of ideas is just as crucial as the information itself (Griffiths, Murdock-Perriera,& Eberhardt, 2023). The potential role of AI and particularly, ChatGPT, cannot be overemphasized in the development of this modern landscape. At Multiverse, we believe that these technologies could revolutionise work-based learning environments by offering effective feedback and positioning the learner as active participants in their feedback process.
A Comparative Study: Traditional Coaching Methods vs AI-feedback
In a recent study (Teasley, 2023), we explored the potential of ChatGPT for delivering meaningful feedback. We compared AI-driven feedback with traditional coaching methods, taking into account the learner’s acceptance and reactions to AI-assisted feedback. We also explored the interaction between both the AI and the learner (Neurerer, et. al, 2018). Although previous research has explored assisted feedback (Maier & Klotz, 2022), this work is at the forefront of using AI to deliver feedback in work-integrated learning environments.
Initial hypotheses were that apprentices would prefer coach feedback over AI-generated feedback. Surprisingly, we found that 70% of apprentices showed a preference for receiving both AI and coach feedback. ChatGPT offered a greater amount of feedback that encouraged self-regulation and autonomy in learners compared to human coaches, showcasing the reliability of AI in providing feedback. Furthermore, we found no significant difference in feedback effectiveness between ChatGPT and coaches, with ChatGPT's feedback slightly favored. Virtual rapport assessments indicated a moderately positive perception of ChatGPT's feedback for its human-like qualities and coherence. Qualitative feedback showed a preference for combining AI's specific and objective feedback with the personal touch and context understanding of human coaches.
Our study suggests that while AI can offer specific and objective feedback, the nuanced understanding and personal engagement provided by human feedback is irreplaceable, advocating for a complementary use of both AI and human feedback in educational contexts. At Multiverse, our exceptional coaches are at the heart of our learning experience. They offer personalized engagement through direct human interaction, which is enhanced by the use of AI technology. Our learners also have access to real-time AI feedback, whenever they need it, through our new on-demand AI tutor. This provides our learners with the tools and resources to reap the benefits from both human and AI feedback.
In summary, by utilizing the cognitive apprenticeship model and AI-enablement (Amankwatia, 2023) we can offer real-time coaching, adaptively scaffold support based on learner performance, and encourage reflective practice through dialogue, enhancing understanding and skill acquisition in a collaborative learning environment.
Study Details
Our study was a mixed methods design which used data from thirteen apprentices enrolled in a technology consulting degree programme. Naturally occurring coach feedback was compared with ChatGPT-generated feedback. This feedback was generated and coded against an Agentic Feedback taxonomy. Surveys measuring apprentice perceptions of feedback, acceptance, motivation, and virtual rapport were developed from the Feedback in Learning Scale (FLS; Jellicoe & Forsythe, 2019).
Survey data was compared and differences were tested for significance and effect sizes. Qualitative data was analysed for key themes and reported. Inter-coder reliability was calculated for feedback coding trials (overall agreement, 79.8%).
Overall, the study demonstrated that ChatGPT's feedback on digital apprenticeship assignments matched the agentic quality of coach feedback and suggests the potential for AI tools to enhance feedback in work-integrated learning by complementing human inputs with timely, specific, and effective feedback (Teasley, 2023).
References
- Amankwatia, T. (2023). Using AI with Cognitive Apprenticeship Theory, Upscaling and Retooling. The Evolllution. Retrieved March 22, 2024, from https://evolllution.com/technology/tech-tools-and-resources/using-ai-and-cognitive-apprenticeships-to-upskill-and-retool-adult-learners(opens new window)
- Griffiths, C. M., Murdock-Perriera, L., & Eberhardt, J. L. (2023). “Can you tell me more about this?”: Agentic written feedback, teacher expectations, and student learning. Contemporary Educational Psychology, 73, 102145.
- Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.
- Jellicoe, M., & Forsythe, A. (2019, August). The development and validation of the Feedback in Learning Scale (FLS). In Frontiers in Education (Vol. 4, p. 84). Frontiers Media SA.
- Maier, U., & Klotz, C. (2022). Personalized feedback in digital learning environments: Classification framework and literature review. Computers and Education: Artificial Intelligence, 3, 100080.
- Neururer, M., Schlögl, S., Brinkschulte, L., & Groth, A. (2018). Perceptions on authenticity in chat bots. Multimodal Technologies and Interaction, 2(3), 60.
- Teasley, W.P. (2023). Evaluating the suitability of ChatGPT to deliver effective feedback in work-integrated learning environments. (Unpublished master's thesis). University of Oxford, Oxford. http://dx.doi.org/10.13140/RG.2.2.26629.77287(opens new window)
Team Multiverse