The Assessment Engine: A Question Bank That Survives Randomization
Designing quizzes that vary, grade themselves, and resist cheating, without turning the experience adversarial.
A quiz on a learning platform is not a form. It is data from which a decision is derived: do they pass, do they get a certificate? That makes its architecture closer to a financial system than to a survey.
Separate the question from the quiz
The first mistake is storing questions inside the quiz. Make the question a standalone entity with tags and a difficulty level, and the quiz a recipe that draws from it: "five easy and three medium from this topic". Every attempt becomes a different variant for free, and you can improve a question without touching the quizzes that use it.
Store the whole attempt, not just the score
- The questions actually shown and their order — the quiz is randomized and you will not be able to reconstruct it later.
- The learner's answer exactly as entered, not as interpreted.
- The answer key in force at the time, because fixing a bad question later must not silently rewrite old results.
- Time taken, which is both the best difficulty signal and the first evidence of cheating.
Auto-grading and its limits
Multiple choice and true/false grade themselves without dispute. Short answers need fuzzy matching and a synonym list, and will sometimes be wrong. Essays cannot be graded automatically with enough confidence for a certification decision — keep a human in that loop and design for it rather than pretending otherwise.
Auto-grading that is wrong 3% of the time looks excellent until that 3% is the difference between a certificate and a failure.
Integrity: raise the cost, don't assume bad faith
- Randomize question and option order per attempt — the cheapest measure and the most effective.
- Set a time limit that prevents looking up every question without punishing slow readers.
- Cap attempts and compute the final score by a rule announced in advance.
- Watch for anomalies — implausibly fast completion, identical answer patterns — as a trigger for human review, not as an automatic verdict.
Camera proctoring and browser lockdown suit formal high-stakes exams. In professional training they usually cost you more in learner trust than they save in integrity.
Questions need data about themselves
After a few hundred attempts, compute a correct-answer rate and average time per question. A question everyone answers correctly measures nothing; one everyone fails is usually badly worded rather than hard. That data improves your bank on its own.
If you want a standard format for the question bank rather than a proprietary one that locks you in, 1EdTech's QTI specification is the established reference for representing questions, attempts and results portably. If you are building an assessment engine inside your own product, talk to us.
FAQ
How many questions does a bank need?
At least three times the quiz length per topic, otherwise randomization is cosmetic and questions repeat.
Should I reveal correct answers afterwards?
Give feedback without exposing the full key if further attempts are allowed, or your bank will be copied within days.
What about a bad question found later?
Fix it for new attempts and rescore old ones through an announced decision rather than a silent edit.