AI companies have made obscene amounts of money deploying tools that have fundamentally disrupted education. They have not really changed how students learn, how teachers teach, but they have changed what academic integrity means. They have done this without much consultation, without adequate research, and without taking responsibility for the chaos they have created.
And now they are selling us solutions to the problems they caused.
Students deserve better. Educators deserve better. And it is time we started demanding it.
These tools perpetuate educational myths because they are trained on the internet, and the internet is full of bad pedagogy.
Tools That Undermine Learning
Companies have shipped mediocre products to be used with millions of students and called it innovation.
Sycophancy by design. These tools are trained to agree with users, to validate their ideas, to tell them what they want to hear. This is a core feature, not a bug. It increases engagement, keeps people coming back, generates the data these companies need. But for learning? Sycophancy is poison. Education requires challenge, discomfort, the friction of encountering ideas that do not confirm what you already believe. AI tools that tell every teacher and student their thoughts are brilliant are not tutors. They are enablers. At worst, it leads to AI psychosis among the vulnerable. More on that later.
Bias as a feature. These tools generate content that reflects the biases in their training data, and companies have known this from the beginning. They put these tools in the hands of teachers and students. Students learning about history, about science, about themselves and their place in the world—knowing that the outputs would be skewed, stereotyped, and in many cases flatly wrong. You can easily get different results if you tell the tools you are in different locations. They will reflect the stereotypes of those locations. Try it. Vendors of these products act like educators should be grateful for the opportunity to be unpaid testers of their flawed products.
Debunked theories treated as fact. I have watched AI tools confidently recommend instructional strategies based on learning styles (a theory that has been debunked for decades). These tools perpetuate educational myths because they are trained on the internet, and the internet is full of bad pedagogy. When a student or a new teacher asks an AI for advice on how to teach or learn, they get a confident response that may be grounded in nothing resembling research-based practice. The companies know their tools make things up. They encourage educators to use them anyway.
They Sold Detection Tools They Knew Did Not Work
When the academic integrity crisis hit, AI companies saw another opportunity. They rushed detection tools to market, promising to solve the very problem their generative tools had created.
These detection tools do not work reliably. The companies know this. Researchers have documented the high false positive rates, the bias against non-native English speakers, the fundamental impossibility of definitively distinguishing AI-generated text from human writing as the models improve. And yet these tools are being sold to schools and universities, being used to accuse students of cheating, being treated as evidence in academic integrity proceedings.
Students have been wrongly accused. Reputations have been damaged. Trust has been destroyed. And the companies that sold these tools have faced no consequences.
This is not innovation. This is profiteering from a crisis of their own making.
They Are Harming Vulnerable Minds
The harms extend beyond classrooms. There is growing evidence of what some are calling AI-induced psychosis—the emergence or worsening of delusional thinking in vulnerable users who engage intensively with AI chatbots.
It is happening now. Users, often isolated or struggling with mental health, are developing beliefs that AI systems are sentient, that they have special relationships with the AI, that the chatbot understands them in ways other humans cannot. The sycophancy that makes these tools engaging also makes them dangerous. They validate beliefs that should be challenged, create feedback loops that reinforce delusions, simulate intimacy and empathy in ways that can be deeply destabilizing for vulnerable people.
Young people are among the most vulnerable. Adolescents who are socially isolated, who are struggling with identity and belonging, who spend hours in conversation with AI systems that affirm everything they say. These are the users most at risk. And these are exactly the users schools serve.
AI companies know about this. They have the data. They can see the patterns of use that correlate with psychological harm. And yet the tools remain designed to maximize engagement, to keep users coming back, to simulate the kind of connection that lonely people crave. I saw a group of educators in an AI training fawn all over a video in which a young elementary age student gave her AI companion a name and treated it like it was her friend. They thought it was cute.
Where is the real-time risk monitoring? Where are the interventions when usage patterns suggest a user is spiraling? Where is the transparency about these harms?
Nowhere. Because addressing these issues would reduce engagement, and engagement is how these companies make money.
What They Owe Us
Here is what I think AI companies owe education (at minimum).
Transparency about limitations. Every AI tool used in education should come with clear, honest documentation of what it cannot do. Not buried in terms of service. Not some warning in small text that responses “may contain bias.” Front and center, in plain language: here is where this tool fails, here is where it is biased, here is what it gets wrong. Educators cannot make informed decisions about tools when the companies selling them obscure the risks.
Education-specific design. Tools built for productivity are not automatically appropriate for learning. Learning requires struggle, error, feedback that challenges rather than affirms. If AI companies want to serve education, they need to build tools designed for pedagogy. Tools that know when to push back, when to ask questions rather than give answers, when to let a student struggle productively rather than providing instant solutions. This is not impossible. It is just not profitable in the same way sycophantic engagement machines are profitable. They aren’t the one-click solutions that Google trainers push with Gemini.
Accountability for harms. When detection tools falsely accuse students, there should be consequences. When AI systems contribute to psychological harm in vulnerable users, there should be consequences. When tools perpetuate debunked educational theories and bad pedagogy, there should be consequences. Right now, there are none. Companies ship whatever they want, collect their money, and leave educators and families to deal with the fallout.
Research-grounded outputs. AI tools used in educational contexts should be trained to recognize and defer to established research. When a user asks about learning styles, the tool should know that this theory has been debunked and say so. When a user asks for teaching strategies, the tool should draw on evidence-based practices, not whatever pseudoscience happens to be well-represented in its training data. This requires investment and intentionality. It requires companies to care about accuracy as much as they care about engagement.
What Educators Should Demand
I do not expect AI companies to do the right thing out of the kindness of their hearts. They are businesses. They respond to incentives. So educators need to change the incentives.
Stop accepting tools without documentation. When a vendor pitches an AI product, ask for bias audits the same way we might ask for a demonstration of accessibility features (we’re doing that, right?) . Ask for the accuracy data. Ask for the research on learning outcomes. If they cannot provide it, they are asking you to experiment on your students with an untested product. That should be disqualifying.
Demand transparency in contracts. When districts purchase AI tools, the contracts should require ongoing transparency about changes, limitations, and known issues. Companies should not be able to update their models and change how tools work without informing the educators who rely on them.
Advocate for regulation. The education market is valuable to AI companies. If we collectively demand standards (through professional organizations, through state education agencies, through federal policy) companies will meet those standards or lose access to the market. We have more power than we realize, but only if we use it.
Name the harms publicly. When AI tools fail our students, say so. Loudly. Companies care about bad press and will often jump to address a publicized response that is problematic. Educators who document and publicize harms are doing a service to everyone who comes after them.
We Deserve Better
Even though I have enormous concerns and reservations about the default use of these tools, I still use them. I see their potential. I have written about how they can support learners and extend what educators can do.
But potential is not the same as reality. And the reality right now is that AI companies have treated education as a market to exploit rather than a mission to serve. They have shipped flawed products, profited from the disruption, and left us to figure out the consequences.
We did not ask for this. We did not consent to being beta testers for tools that affect our students’ learning, mental health, and futures. We deserve partners who take education seriously. Who build with pedagogy in mind, who are honest about limitations, who take responsibility when things go wrong.
Until AI companies start acting like partners instead of profiteers, educators should treat them with exactly the skepticism they have earned.