There is a lot of value in being able to create and edit your own digital curriculum regardless of school setting. I have witnessed examples where teachers were asked to use vendor curriculum they weren’t allowed to edit. Some of those lessons were offensive, including lessons about slavery and about Deaf culture. I complained. I have no confidence anything changed.
I had always assumed somebody reviewed those lessons before they went out. If they did, they went out like that anyway.
I think about those situations a lot. Quite a bit of my time is spent building courses and providing guidance to educators/instructional designers on how they can do it too. Most curriculum gets a review before it reaches students, and that one (hopefully more) reader is asked to catch everything at once: accessibility, accuracy, assessment design, language support, cultural framing, and what it’s like to actually do the work as a student. I’ve been that reader. You notice what you were trained to notice, or what your experience has taught you should be noticed. An accessibility specialist sees the heading structure, design, color contrast, etc. A content expert sees a craft term defined three different ways, or framing that might confuse a learner. A classroom teacher sees that the time estimates don’t include twenty minutes of required reading. Few people, if anyone, can see all of it.
AI makes this harder. It writes fluent, confident course material fast, and that material reproduces the defaults: the average student, the standard route through an assignment, the familiar example. And, as I’m fond of saying, the default sucks. When AI-written pages go out without real review, the default shows up on every page.
What I did with my own course
This fall I put the fourth version of my Creative Writing Through D&D course (six units, high school) through a review process I’ve been building. I call it layered review. Instead of one pass, the course goes through a fixed sequence of layers. Each layer covers one set of concerns, and its reviewers read what the earlier layers found and look for what they missed.
- Scripts run first: accessibility markup against WCAG 2.2 AA, reading level, and the answer choices on every quiz.
- Learner access: English-learner support, speech and language, gifted and twice-exceptional learners, accessibility, culturally sustaining teaching, and social-emotional and trauma-informed practice.
- Content accuracy: craft terms, literary facts, game rules.
- Teaching and assessment design: does every rubric row score what the task asks, and do the time estimates add up?
- Simulated students work through the revised pages and log every place they get stuck.
- Specialists judge what the students found.
- An AI model from a different developer checks the final pages for accuracy, and a second outside model reads them as a skeptic.
- Meaning checks after every edit, and a consistency pass across the whole course at the end.
The reviewers are AI-run, each built from a written profile of an expert in a field and that field’s research. I direct the whole thing and make the calls no reviewer should make.
The quiz check found a unit where the correct answer was the longest choice on six of seven questions, and the same letter on six of seven. A student could score well by picking the longest answer. I also found alt text that gave away the answer.
The learner access layer found a quiz a screen reader user couldn’t complete because its form controls had no labels. It also found a discussion question that would have turned an honest answer from a student living with violence into a disclosure in front of classmates. The content layer found the climax of a story defined three different ways on three pages. The assessment layer found a sentence frame that, filled in exactly as written, earned the lowest level on the rubric it pointed to.
The simulated students
I organized the review around the students that course writers tend to miss when we write for the default. The simulated roster includes an English learner who drafts key moments in her home language, a long-term English learner who works more than twenty hours a week and does his coursework on a phone, a Deaf student whose first language is ASL, a blind student who uses a screen reader, a student with dyslexia who composes by voice typing, students who spend ten or more hours a week caring for family… The list goes on. Each profile is built from published research on that group of learners. Each student’s week is drawn too. Some weeks are calm. Some include a shift change or a sick sibling.
They complete each assignment the way that student would. One wrote the key moment of her scene in Dari, as the page invited her to, and a teacher reading only the English would have scored her work a 2 out of 4, because the rubric never said how to read a home-language passage. Another never found the spoken-response option, which sat about 2,000 words down the page. The blind student found a response route only because he switched his screen reader from heading navigation to reading everything; the route was labeled in bold instead of a heading. The student using voice typing found that her dictation software produced no quotation marks, and gave up halfway through adding them by hand to a dialogue assignment. The Deaf student reread the signed-response directions three times and then submitted both a video and a written version, just to be safe.
That last one gets me. A student did twice the work because we weren’t clear.
These are predictions, and they’re labeled as predictions everywhere they appear. They are real things that could happen though. Of course they don’t replace a pilot with real learners. But they catch things a reader skimming the page won’t, because they actually do the work. And when the specialists judged what the students found, they kept looking at the same pages. Across five units, they proposed 636 fixes, and 300 of them were problems no student had raised.
Where I decide, and how each layer is checked
Some calls belong to a person. When reviewers proposed adding historical context to a mentor text whose hero fought for the Confederacy, I decided the course wouldn’t refer to the Confederacy at all and replaced the text. I ruled out disability metaphors in course language, meaning phrases that use a disability to stand for ignorance, carelessness or fear. The course had a “blind guess,” “blind spots,” “blind obedience,” a “blind” peer review and “decision paralysis,” and it used stammering as the sign that a character was afraid. Those became plain words: a guess, unquestioning obedience, a character who trips over their words. Literal references to blind, Deaf and disabled people stayed. Neither the accessibility reviewer nor the simulated blind student flagged those phrases; the outside model did. I set limits on what the student profiles record. No review layer should be making those decisions, and every other finding gets a ruling with a written reason, kept in the course records.
Every layer also gets checked by something other than itself, me included. The outside accuracy check found answer keys that rejected correct answers. In one unit, more than half of the passages it flagged were text the review itself had written.
A privacy sentence was copied across the course to make the wording consistent. Once it was everywhere, it implied that a student’s statement about self-harm would stay private, while every assignment’s reporting notice correctly said it would not. A meaning check that compares every edited passage against the pages around it caught it before it reached a student.
Fixing the next course before it’s written
Nearly every kind of problem showed up in every unit, even after earlier passes had fixed individual cases. Correcting them one at a time didn’t stop new ones, because each new page was written the way the earlier ones had been. So I turned the findings into 25 rules for the first draft of the next course. Put every response route near the top of the page and say how each one is scored. Write a model for every route and length you offer. Balance quiz answer choices when the quiz is written. Don’t write a criterion that requires sight or hearing. Some of them reversed defaults built into my own course-building tools, including one that kept home-language permissions off student pages. The tools follow the new rules now.
I didn’t time the review. In my experience, it goes much faster than a team of people reviewing separately, sharing what they found and then making revisions. AI-run reviewers aren’t certified experts, and they don’t replace a certified accessibility audit, a district’s own review process, or the teachers who will teach the course. Every layer missed things a later layer caught, so no single layer, including the last one, should be treated as the whole review. I think the layered review is worth doing for one reason above all: when human reviewers get the course, it’s much closer to finished than anything they’ve seen before. Then that time can go to the few things that are left and to what’s next. What could we do if that first pass came off the workload? What could we add to a course, like interactives, multiple pathways or enrichment, if we had a little more time?
Questions to ask the people who made a course
- Which groups of learners was the course reviewed for?
- Who or what checked each reviewer’s work, and is that check independent of whoever wrote the pages?
- Were students, real or simulated, asked to complete the work as well as read it?
- Were the answer keys and scored samples checked against a source other than their author?
- When a sentence was corrected in one place, how do you know every copy of it changed?
Then ask to see the records behind the answers.
I’d like the next learner who opens a course, whether that’s a Deaf student, a student doing homework on a phone after a shift, or a student writing in their home language, to experience one that was checked with them in mind.
How I can help
- Review a course before it reaches students. If you’re about to adopt or publish a course, I can run it through layered review and give you the findings, the fixes and the records behind every ruling. You decide what changes.
- Teach your team to run its own review. In the AI Course Design Cohort, I work with your course developers over about two months as they write with AI one review pass at a time, so the checks become part of how your team builds.
- Find out where to start. The AI Readiness Diagnostic is a few conversations with your leadership and a look at how courses get written and reviewed now, followed by written recommendations in priority order.
- Bring the questions to your staff. I run workshops and keynotes on reviewing AI-written material, in person or virtual, for a PD day or a conference session.
I also wrote up the full method in a longer paper, which I share with the administrators and curriculum teams I’m working with. If that might be you, book a time to talk about your courses.
