If you've ever tried using a generic AI tool to practice Specialized High School Admissions Test problems with your child, you've probably hit this wall: sometimes the answer it gives you is just wrong.
Not wrong in a subtle way. Wrong in the way that teaches your child a bad habit — they read the explanation, it sounds reasonable, they move on, and now they have the wrong mental model locked in.
This is the biggest unsolved problem in AI-powered SHSAT prep for NYC students. We've been obsessing over it for months. Here's what we found and what we built to fix it.
The Real Problem With AI-Generated SHSAT Questions
Most people assume AI generates bad questions. That's not actually what happens.
The questions themselves are usually fine. The problem is the answer key.
When a language model generates an SHSAT practice question, it produces the question, four answer choices, and a "correct answer" label — all in one pass. The model doesn't stop and independently solve the problem. It just assigns a label based on pattern matching from its training data.
For straightforward questions, this works. But for the specific question types on the Specialized High School Admissions Test — ratio word problems, main idea passages, grammar revision — the model frequently labels the wrong option as correct.
Your child reads the explanation, nods along, and learns the wrong answer. That's not test prep. That's anti-preparation.
What We Built to Fix It
We added a two-pass verification system to every question generated for NYC SHSAT practice.
Pass 1: The model generates the question and labels an answer.
Pass 2: A second model call is made — independently, with no knowledge of what the first pass said. It's asked to solve the question from scratch and return which option it believes is correct.
If the two passes disagree, the question is flagged. It never reaches a student.
This matters especially for ELA. The Specialized High School Admissions Test ELA section has very specific question types — central idea inference, revision/editing with numbered sentences, author's purpose — where general AI models routinely get confused. We built explicit rules for each type:
- Central idea answers must cover the whole passage, not just the last paragraph
- Inference answers must be supported by the text, not just plausible-sounding
- Revision/editing questions must use the actual numbered-sentence format from the real SHSAT
The Human Layer
Algorithmic verification catches the most obvious errors. But SHSAT prep for New York City students requires a higher standard.
Every question that passes the two-pass check goes into a review queue. A human reviews it before it enters the live question bank. Each question is either approved or rejected.
Rejected questions are permanently removed. They're never shown to students, even if the AI "thought" they were correct.
This means every practice question your child sees on SHSATlab has been independently verified by a second AI check AND reviewed by a human. That's a standard that prep books — which have been printing the same errors for years — can't match.
What This Means for Your Child's SHSAT Prep
The students who score highest on the Specialized High School Admissions Test aren't necessarily the smartest. They're the ones who've built accurate mental models through accurate practice.
Every wrong question a student practices against is a debt they'll pay on test day. They've drilled a wrong pattern deep enough that it activates automatically under time pressure.
The fix isn't to practice more. It's to practice on questions you can trust.
For NYC families preparing for the SHSAT, that distinction matters enormously. The difference between a 550 and a 580 — the difference between Brooklyn Tech and Stuyvesant — is often just a handful of topics where a student has a slightly wrong mental model. Fix the model, the score moves.
Why Most AI Tools Don't Do This
The two-pass system costs more per question to run. The admin review costs time. Neither of these add up to a meaningful revenue line for a general-purpose AI company.
We're not a general-purpose AI company. We're a SHSAT prep platform for New York City students. The whole product stands or falls on question accuracy. So we absorbed the cost.
This is the standard we hold ourselves to — and the reason we built a question bank review system before we built anything else.
If you want to see it in action, join the waitlist. Early access families are our first test, and we want every question they see to be one they can trust.