The model was never the hard part. The hard part was deciding what it's allowed to do alone, and building a screen where a person can check its work quickly enough that checking is worth it.
The problem
We collect subject requirements from a lot of web pages. One page says "A-level Mathematics", another says "Maths at A2", a third buries it in a paragraph about something else. Our editors had to read every page, work out what it meant, and map it to our master list of subjects. It was careful work and there was far too much of it.
An AI can read those pages faster than any of us. But a wrong subject on a published record does real damage, so "the AI said so" can't be the last step.
How a review works
Pick a model and run the shared prompt
The editor chooses a model for the run, from lighter to heavier: Gemini, Gemma or DeepSeek. Each model shows its daily limit, so nobody burns through a quota by accident. The prompt is shared across the team and versioned. Anyone can see who changed it last and when. If one page needs a tweak, you can adjust it, and that change is on the record too.
Check the draft against the source
The AI gives back two things: a short text summary and a structured table. Both are interactive. The source page sits beside them with the relevant text highlighted, so the editor isn't hunting for where a subject came from. Any cell in the table can be edited directly, with validation and undo. When it looks right, Save & Verify.
Map to the master list, with one hard rule
A subject identifier then maps every subject the AI found to our master list. If it's an exact match, a score of 1.0, it's ready to publish on its own. Anything less shows its closest match and a similarity score, and a person picks. "Further Maths" might sit close to "Mathematics", but close isn't the same, and that's exactly the call I want a person to make.
The similarity score is plain string similarity. I'm careful not to call it semantic matching, because it isn't, and that's fine: its only job is to suggest a candidate to a person.
The parts around the AI
- Bulk reruns with status tracking.
- Per-person and per-source stats, so we can see where the work is.
- Token cost per run for DeepSeek, so cost is something we look at, not something we find out about.
What I'd keep
Separate what the AI drafts from what it may decide. In this tool it drafts everything and decides almost nothing. The one thing it may decide, an exact match, is the one thing it can't get subtly wrong. I go into that rule in more depth in LLM-assisted data validation: what the AI is allowed to do alone.
Stack
React with TypeScript, Node.js and Express, MongoDB, Gemini, Gemma and DeepSeek models.
My role: I designed the review workflow and where the AI hands over to a person. *The 60-70% reduction in manual QC is measured across the Subject Validator and the Fees Validator together.