The Effect of AI Hint Transparency on User Retention in an Educational Platform

Authors

  • Maria Zinchenko Graduate School of Business, HSE University
  • Konstantin Bagrationi `Graduate School of Business, HSE University
  • Olga Tunkevichus Graduate School of Business, HSE University

DOI:

https://doi.org/10.34190/icer.3.1.5559

Keywords:

artificial intelligence in education, AI transparency, learner retention, programming education, learning analytics, instructional design

Abstract

Generative artificial intelligence (AI) tools are increasingly embedded in coding courses, but their educational value depends on how this assistance is delivered to the student (Brown et al., 2025). This study tests whether revealing that clues are generated by AI - indicating their origin, naming their limitations, and requiring verification - affects the early retention of teenagers in an online course on coding. Developed a dedicated web platform providing a two-week self-paced Python introductory course, and conducted a supervised field experiment involving 93 high school and college students which were divided into three transparency conditions by means of a fixed alternating sequence: clues without mentioning that they are generated by AI (A), indicated clues with the verification instruction (B) and indicated clues with the verification instruction and the mandatory microphlexy (C) Four behavioral metrics were also calculated: week-2 retention, module-1 completion rate, stuck rate, and hint-to-progress time. The more transparent formats were associated with significantly higher retention and completion rates, as well as lower trapping frequency and faster progress after the hint, with the most complete C format having the strongest effect. The results suggest that the tooltip design, which combines source disclosure, information about AI failure, test queries and microflexy, is associated with lower early drop-off without changing the content of the course. This can be seen as a component of the measurable design mechanism.

Downloads

Published

2026-11-13