Inside PenLeap: Grading 9,200 Class 10 Essays Daily Without Hallucinated Marks

On a normal August 2025 weekday, PenLeap — our in-house AI writing and exam-prep product for students 11+ — grades about 9,200 CBSE Class 10 English essays. The number that keeps us up at night is not throughput; it is trust. A single hallucinated mark — a 6/10 the model cannot justify against the rubric — and a teacher stops believing every score after it. So we built grading to be boring and auditable: every mark is locked to a rubric band and anchored to a quoted line from the student's own essay. This post is the architecture: rubric-locking, the double-pass eval, and the teacher-override loop that closes the gap. ## The Answer in 60 Words PenLeap never asks a model for a free-form mark. It scores each rubric criterion independently, forces the model to quote the exact line that justifies each band, runs the whole essay twice and adjudicates disagreements, and routes any low-confidence or wide-disagreement essay to a teacher. The teacher's correction feeds back into the rubric prompts. The result: roughly 0.84 quadratic-weighted-kappa agreement with human raters and no unexplained marks. ## Why This Matters Now (Aug 2025) CBSE Class 10 board prep peaks from August onward, and English writing is where students lose the most marks. Teachers want AI grading speed but won't accept AI grading they can't audit. The research backs the caution: a synthesis of 65 studies (Jan 2022 to Aug 2025) found LLM-human agreement ranges widely, mostly 0.30 to 0.80, with models showing rubric-interpretation drift, verbosity bias, and scale misalignment (arXiv 2512.14561). The fix is not a bigger model. It is structure: locked rubrics and evidence-anchored scoring measurably stabilize LLM evaluation (RULERS, arXiv 2601.08654). ## What Is a Hallucinated Mark? (And Why It Kills Trust) A hallucinated mark is a score the model produces that it cannot tie back to the rubric or the student's text. The model says 7/10 for "coherence," but if you ask why, it invents a justification that does not match what the student wrote. Free-form LLM grading does this constantly because the model optimizes for a plausible-sounding number, not a defensible one. Fine-tuned models can reach very high reliability — one study reported an intraclass correlation of 0.972 for a fine-tuned ChatGPT on EFL essays (Yavuz, 2025, BJET) — but reliability collapses on analytic rubrics if you let the model freestyle. We removed the freestyle. ## The Cost of a Wrong Mark Is Not Symmetric A grading engine can fail two ways, and they are not equally bad. A false-low mark — the model scores a good essay too harshly — frustrates a student and gets corrected when they complain. A false-high mark — the model praises a weak essay — is worse, because the student never learns they have a problem and walks into the board exam confident and wrong. So we tune the escalation thresholds asymmetrically: we are quicker to escalate an essay the model rates highly but cannot anchor with strong evidence than one it rates poorly. A confident high mark with thin evidence is exactly the pattern a hallucination produces, and it is the pattern that does the most damage to a student. Most grading systems treat all disagreements the same. We do not, because the downside is not the same. This is the kind of judgment call that does not show up in an accuracy number but decides whether teachers trust the system after a month of use. ## How PenLeap Grades One Essay (Step by Step) ## The Double-Pass Eval: Why Two Passes Beat One Big Model Verbosity and position bias are real: models reward longer answers and over-weight what they read first (arXiv 2512.14561). Two independent passes with evidence anchoring catch most of this because a hallucinated band rarely survives being asked for a verbatim quote twice. We are not claiming perfection — strong open models land around 0.6 quadratic-weighted-kappa on single-score grading before structure (research synthesis). Rubric-locking plus the double pass is what lifts our criterion-level agreement to roughly 0.84 against our teacher panel. ## The Teacher-Override Loop (The Part That Compounds) The override loop is the difference between a model that drifts and one that improves. Every teacher override is logged against the exact rubric criterion, with the model's evidence and the teacher's reason. Once a week we cluster the overrides and tighten the band descriptors or add a counter-example to the prompt for the criterion that drew the most corrections. Overrides on Grammar dropped from 6.2% to 2.4% across August as we hardened the SPAG descriptors. This is the same evidence-anchored discipline behind our PenLeap Hindi essay grading engine. ## Why the Student Sees the Evidence, Not Just the Mark A mark with no explanation teaches nothing. When a Class 10 student gets back "Content: 3/4", they learn that a number happened to them. When they get "Content: Band 3 — you addressed the prompt but your second paragraph drifts off-topic" with the exact sentence quoted, they learn what to fix. So the student-facing screen leads with the evidence, not the number. Each criterion shows the band, the quoted line from their own essay, and one concrete next step — never more than one, because a student who gets six corrections fixes none. This is the same feedback philosophy behind the rest of PenLeap: the score is the start of the lesson, not the end of it. Teachers told us this single design choice changed how students reacted to grades. A bare number invites an argument; an evidence-anchored mark invites a revision. The student cannot dispute that their own sentence drifts off-topic when the drift is quoted back at them, so the conversation moves straight to "how do I fix it" instead of "why did I lose a mark". That shift — from grievance to revision — is the entire point of grading practice essays at all. ## Quality-Gate Checklist for AI Grading ## Where 9,200 Daily Essays Actually Go Not every essay takes the same path. The whole design goal is to let AI handle the confident majority and reserve scarce teacher time for the cases that need a human. Here is how a typical weekday's 9,200 essays distribute across the pipeline. That distribution is the business case. A teacher who used to mark 30 essays a night now reviews the roughly 290 escalations spread across the whole school, plus any overrides they choose to make. The throughput comes from trusting the model exactly as far as its evidence justifies — and no further. ## What the Grading Prompt Actually Contains People ask what the prompt looks like. It is not clever wording — it is structure. Each criterion gets its own call with four fixed parts, and that rigidity is the point. A loose prompt is how you get rubric drift across 9,200 essays.

Prompt part	What it pins down	Why it stops hallucination
The band ladder	Every band with its exact descriptor and fixed mark, verbatim from the CBSE rubric.	The model selects from a closed set — it cannot invent a half-mark or a band that does not exist.
The evidence rule	"Quote the exact sentence from the essay that justifies your band. If you cannot, return INSUFFICIENT_EVIDENCE."	Forces the mark to be traceable to the student's own words, not the model's imagination.
Two worked examples	One essay scored at a low band, one at a high band, each with its evidence quote.	Anchors the model's interpretation of each band so it matches the teacher panel's, reducing drift.
The output contract	Strict JSON: band id, mark, evidence quote, one improvement tip. Anything else is rejected.	A malformed or free-form response never reaches a student — it is caught and retried at the parser.

The two worked examples matter more than anything else. Research on locked rubrics and evidence-anchored scoring shows that pinning the model's interpretation of each band is what stabilizes its output across thousands of essays (RULERS, arXiv 2601.08654). Without the examples, "Band 3 content" means slightly different things on essay 1 and essay 9,000. With them, it means the same thing every time. ## When Not to Use AI Grading At All If the assessment is high-stakes and final — the actual board exam, a scholarship decision, a pass/fail gate — do not let AI be the last word. Use it as a fast first read that a human signs off. AI grading shines for daily practice at volume, where instant feedback is worth more than perfect marks and a student writes ten essays instead of one. It is the wrong tool when a single number changes a child's future. We are explicit with schools about this line, and we draw it on purpose. PenLeap grades practice; teachers own the verdict. ## Real Example: A Jaipur CBSE School, 1,400 Students A Jaipur CBSE school put its Class 10 cohort — about 1,400 students — on PenLeap for daily English writing practice in August 2025. Before, each student got teacher feedback on roughly one essay a week because marking is slow. After, students averaged five graded essays a week, with teachers spending their time only on the 3% PenLeap escalated and on overrides. Teacher marking hours per week fell while feedback volume rose 5x. The adaptive question side of this — why two students see different prompts — is in our post on PenLeap's adaptive difficulty engine, and the generation side is in our CBSE pre-board question generator. We build these rubric-scoring engines for clients through our AI & automation team, often alongside custom web platforms. As Hrishikesh, our CTO, says: a mark a teacher cannot audit is not a feature, it is a liability. ## FAQ ### How does PenLeap stop the AI from hallucinating marks? Every mark is locked to a pre-written rubric band and must be backed by a verbatim quote from the student's own essay. If the model claims a band without quoting the line that justifies it, the response is rejected and retried. No evidence, no mark. ### What is rubric-locked grading? Instead of asking the model for a free-form score, you give it the exact rubric bands with fixed marks and force it to choose one per criterion. It cannot invent a number like 7.5. This stabilizes scoring and removes the rubric-interpretation drift that plagues free-form LLM grading. ### Why grade each criterion separately instead of one overall score? Analytic, per-criterion grading stops a weakness in one area from contaminating an unrelated score, and it gives students actionable feedback. Research shows single-score free-form grading is where LLM-human agreement drops, so PenLeap grades Content, Coherence, Grammar, and Vocabulary independently. ### How accurate is PenLeap compared to human teachers? Criterion-level agreement with our teacher panel runs around 0.84 quadratic-weighted-kappa after rubric-locking and the double-pass eval. Unstructured LLM grading typically lands around 0.6 on single-score grading, which is why the structure matters more than model size. ### What happens to essays the AI is unsure about? About 3.1% of essays escalate to a human teacher — anything with a wide disagreement between passes, low adjudication confidence, or that is off-topic, very short, or heavily code-mixed. AI handles the confident majority; teachers own the hard cases and the final verdict. ### Can Softechinfra build a rubric-scoring engine for our institution? Yes. The same evidence-anchored, double-pass architecture works for any rubric-based assessment — essays, viva transcripts, code reviews. Our AI and automation team ships these as fixed-scope engagements. Book a call and we will map it to your rubric.