Skip to main content
AiGORA
Back to Q&A
TextQ&A
6

Why are AI outputs often taken as much more solid than they actually are?

I have noticed that students (and colleagues) often react very uncritically when reading AI outputs, and I wonder why that is. Do they not understand that what they get is just basically the most probable thing the internet has to say on the matter, including all the biases that come with that?

Anon · 10 Nov 2026

Responses from the team

2 perspectives from the community

  • Mirjam Glessmer's profile photo

    Mirjam Glessmer

    Hej! I think you already mention part of the answer: Many people might not actually be aware of how an AI output is produced technically, and therefore perceive it as more magical than what it actually is.

    But AI outputs are also often written in ways that make it easy to fall into the trap of assigning them more credibility than they deserve. Fan et al. (2026) describe four "pitfalls" and how they can be addressed:

    • The Pitfall of Coherence is exactly about uncritically taking on unsupported claims because a text reads so nicely and is structured so convincingly. This can be addressed by making sure that the AI being used does not over-scaffold, but provides “balanced aid”, e.g. only a hint for the next step and not a complete solution, and only if a learner requests it.
    • The Pitfall of Transparency: AI outputs use expert-like framings, references, and step-by-step explanations, which can be misinterpreted, e.g. a guarantee of correctness, when they are not. Avoiding this pitfall requires scaffolded and contextualised verification.
    • The Pitfall of Pandering: It is easy to fall victim to AI’s sycophancy and end up in an echo chamber. This can be prevented through engineered dissent, meaning that the AI provides counterarguments or different perspectives unprompted, or prompts the learner to reflect on what makes them change their mind, when AI agrees with them, etc.
    • The Pitfall of Fluency, mistaking easy to read AI outputs for the reader's own competence and understanding, can be addressed by onloading metacognition in the form of “desirable frictions”, micro-interruptions slowing down the process at key moments to prompt monitoring and evaluation. This could be built into the AI or done by a teacher, e.g. by checking correct understanding of learner intentions, prompting quality-checking of answers, or reminding learners to be critical

    --

    Fan, Y., Yan, L., Molenaar, I., Greiff, S., and Gašević, D. (2026) A metacognitive approach to learning and performance in human-AI interaction. Commun Psychol 4, 150. https://doi.org/10.1038/s44271-026-00548-y

    Read all posts that Mirjam has responded to →

  • Rachel Forsyth's profile photo

    Rachel Forsyth

    Your observations are interesting, because in published research students sometimes report that they need to be careful and critical with GenAI outputs, even when they are using it a lot (e.g. Fošner and Aver (2025) surveying Slovenian students).

    One possible explanation is that this acceptance is an artefact of 'socialisation' of GenAI use. Although we all know it may be unreliable, for various reasons of convenience (always available) and avoidance of more difficult interactions (e.g. with humans like teachers), people are using the GenAI outputs expediently (Pal et al, 2026). Jansen and Meyer (2026) suggest that students see AI as an agent which has performance superior to them across a range of standardised tasks, and compare themselves unfavourably to the AI product.

    It's good that you observe this: it's important to keep challenging people to be critical, and to devise activities for students where it's clear that you value what they as human individuals bring to your course.

    --

    Fošner, A., & Aver, B. (2025). AI chatbots in higher education: Students’ beliefs and concerns. Sustainable Futures, 9, 100734. https://doi.org/https://doi.org/10.1016/j.sftr.2025.100734

    Jansen, T., Meyer, J., Fleckenstein, J., Wigfield, A., & Möller, J. (2025). “Can (A)I do this task?” The role of AI as a socializer of students' self-beliefs of their abilities. Learning and Individual Differences, 122, 102731. https://doi.org/https://doi.org/10.1016/j.lindif.2025.102731

    Pal, O., Duddu, V., Goyal, A., Goel, D., & Saha, K. (2026). Do We Know What They Know We Know? Calibrating Student Trust in AI and Human Responses through Mutual Theory of Mind Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, Barcelona, Spain. https://doi.org/10.1145/3772363.3799386

    Read all posts that Rachel has responded to →

Comments

Share your thoughts — comments are reviewed before they appear

No comments yet. Be the first!