LLM-assisted data validation: what the AI is allowed to do alone

Most "AI for data quality" demos stop at the moment the model returns a nice-looking answer. In production that's where the work starts. The question isn't "can it read the page?", it's "what happens when it reads it slightly wrong?"

Two tools, one set of rules

At Edvoy, in Chennai, we run two AI-assisted validation tools on data scraped from the web, built in React, TypeScript, Node.js and MongoDB. The Subject Validator reads subject requirements and maps them to our master list. The Fees Validator finds fees on fee pages and applies them to the right intakes. Together they cut manual QC by 60-70%.

They look different, but I built both on the same five rules,.

Rule 1: the AI drafts, people decide

Everything the model produces is a draft. In the Subject Validator it's a text summary and a table. In the Fees Validator it's highlights on the source page. Both are things a person reviews, edits and then verifies.

I know that sounds slower than letting the model write straight to the database. It isn't, because the expensive part of manual QC was never typing. It was reading a long page to find the two facts that mattered. The model does the reading. The person does the deciding, in a fraction of the time.

Rule 2: only exact matches skip the queue

There's one thing the AI may decide alone. When a subject it found matches our master list exactly, it publishes. Anything less, even a close match, goes to a person with the closest candidate and a similarity score.

I made the rule strict on purpose. "Further Maths" and "Mathematics" are close as strings and different as requirements. A threshold like "publish above 0.9" would be right most days and quietly wrong on the days it mattered. Exact match is the only bar a model can't clear by being nearly right. I make the same argument for record matching in token sort matching, explained.

Rule 3: show the source next to the answer

A reviewer can only check an answer quickly if the evidence is right there. In both tools the source page sits beside the AI's output, with the relevant text highlighted. In the Fees Validator the summariser highlights every amount and date, and it also flags words like "may" that change what a number means.

Without the highlights, people either re-read the whole page, which wipes out the saving, or they skim and trust the model, which wipes out the point of review.

Rule 4: treat the prompt like code

The prompt in the Subject Validator is shared by the whole team and versioned. Everyone can see who changed it last and when. If one page needs a different instruction, the change is on the record.

A prompt that anyone can edit quietly is a config that anyone can edit quietly. When output quality changes, the first question is "did the prompt change?", and you want to answer it in seconds.

Rule 5: pick the model per run, and watch the cost

Editors choose the model for each run: Gemini 2.5 Flash, a Gemma 3 model or DeepSeek. Each shows its daily limit, and token spend is tracked for DeepSeek. A short, clean page doesn't need the heaviest model. A messy one might.

Making the choice visible did two things for us. It stopped quota surprises, and it made cost something the team sees every day, not something finance asks about later.

And one rule for speed: check once, apply everywhere

One fee often applies to many intakes. Once a reviewer verifies it, one click applies it to the right set: every intake, every full-time intake, or every intake at the same campus. Each record keeps who verified it and when.

The AI saves the reading. The bulk apply saves the retyping. The audit trail keeps both honest.

A short checklist

1. Write down what the model may decide alone. Keep that list short.
2. Make everything else a draft that a person verifies.
3. Put the evidence next to the answer.
4. Version the prompt, and show who changed it.
5. Choose models per job, and keep cost visible.
6. Record who verified what, and when.

Honestly, none of this is about AI. It's the same thing I do with bots: automate the repetition, and put people where the judgement lives.