Why We Redesigned Talent Evaluation: From MCQs to Work Simulations
· 9 min read
Contents
Every company hiring for a role wants to know whether the person can do the work and deliver value in it. They would also like to know if the person will fit the team, stay motivated and be right for the salary. If there were an oracle that told you how a candidate will perform six months after joining, everyone would use it.
There isn’t one, so for the last two decades, hiring at scale has relied on standardised online assessments. Those assessments test proxies for the job, because a computer could only grade answers that had one predetermined right answer. LLMs changed that constraint, and that is why we rebuilt evaluation at Superset from scratch.
Why assessments look the way they do
Assessments exist because interviews don’t scale. A single role can get thousands of applicants. If you want to give all of them a fair chance, you can’t interview everyone.
Running physical exams for that many people was expensive, so exams moved online. At first they were the same pen-and-paper exam, taken on a computer in a college lab.
Delivering the test wasn’t the hard part. Evaluating it was. If thousands of candidates write subjective answers, someone has to read every one of them, and that takes ages. So the format had to be something a computer could grade deterministically and quickly. Computers are very good at checking whether two things are equal, and that is how the MCQ became the default.
Coding tests went the same way. Containerised compilers, and later Docker, let candidates write and run real code during a test. But the check was still binary. Give the function an unsorted array and see if what comes back is sorted.
For a while this worked. Then the questions got simpler to suit the checker: is this number prime, is this string a palindrome. The question bank was small, it converged over time, and candidates learned the answers.
A proxy of a proxy of a proxy
Think about what a customer support associate does. They get a ticket. They read it, understand it, follow an SOP, work out what is going on, ask for more details if they need them, and write a reply. Sometimes it’s a phone call instead of an email.
To test that at scale, the job was broken down into skills, mainly English and logical reasoning. Then each skill was replaced with something a computer could grade. You can’t check someone’s written English against an expected answer, so the test asked about synonyms, antonyms and gerunds instead. Answer twenty of those and you had a verbal ability score.
Every step takes you further from the job. Knowing grammar rules doesn’t tell you whether someone can write an email that actually answers the customer. Aptitude questions on time and work are a decent test of reasoning, but a candidate with good practical sense who is weak at maths problems loses out.
Coding is no different. Someone who has memorised how to reverse a string hasn’t shown that they can find their way around a real codebase.
So the limits of evaluation led to tests built on a proxy of a proxy of a proxy.
Integrity got harder too
Meanwhile the setting changed. Candidates now take tests remotely, on their own laptops. Hiring has become more remote too, since visiting every campus in person is expensive.
That makes integrity more expensive to protect. And a generic question with one right answer is exactly what today’s AI tools can solve in seconds.1 Simulations still run inside proctored delivery. Integrity flags appear on the report with their evidence, for a person to weigh, rather than as an automatic fail.
What changed
LLMs changed the economics of evaluation. An answer no longer has to be reduced to an option or an expected output just because thousands of candidates need to be evaluated. It can be as subjective as the job needs, whether that is writing, speech or code.
That doesn’t mean an LLM can tell you whether a person is good. Someone still has to design the rubric, and that needs domain expertise. The model applies it.
Once evaluation stops being the constraint, the proxies aren’t needed. You can ask the candidate to do what they would do on the job, and evaluate the signals from that.
Simulating the job
That is what we built Skill Studio for. A work simulation puts the candidate in a realistic version of the job and evaluates what they do there. It runs as part of Superset Assessments.
For the support role, the candidate gets a console with live tickets. They read each one, check the SOP, ask for what’s missing, reply, handle an angry follow-up and decide when to escalate.
Software engineering is more interesting. AI can write code now, but there is still a person in front of the machine. That person has to make sense of the business requirement, tell the AI what to build, and debug it when something breaks. The AI can edit a file, but someone has to know which file. It can suggest architectural options, but someone has to pick the better one.
So instead of asking for a palindrome, we give the candidate a repository. It’s hypothetical, but as close to a real one as we can make it, with a database, a front end, logs, tests and a ticket to work on. There is an AI assistant built in, and we want them to use it. How they use it is part of what we evaluate: what they ask it, whether they check what it gives back, and which code they write themselves.
Other roles get their own environments. There is a case interview with a data room, a business to run through a series of decisions, and a production incident to triage from the logs.
How we grade a simulation
A simulation produces far more than an answer sheet. Every query, file opened, message sent and copilot suggestion accepted or rejected goes into an event log. The report is worked out from that log after the candidate finishes.
We don’t hand the whole log to a model and ask for a score. A lot of a simulation can be checked by ordinary code first. Did the candidate finish every step? Did they find the defect we seeded in the drawing? Is their calculation within tolerance? If the attempt is incomplete, the score is capped and the model isn’t called at all.
The model gets a narrower job. It answers specific questions about the evidence, such as whether the candidate checked the copilot’s suggestion before applying it. Every answer has to cite something in the log, like a query, a test run or a message. If it can’t point to anything, the answer counts as no evidence. We call this citation-or-zero.
After that, fixed rules limit what the model can award. A competency can’t be rated strong unless the signals it depends on are in the log, and some failures cap it outright. If a candidate applies a wrong copilot suggestion without checking it, their rating for AI use is capped, however good the rest of the work is.
The report recommends advance, review or reject, and a recruiter confirms the decision. Each score on it links back to the point in the log it came from.
Keeping the environment realistic
Real work is messy, so the simulations are too. The data includes decoys that look like the cause but aren’t. The data room only opens an exhibit when the candidate asks the question it answers. The copilot is useful, but sometimes deliberately wrong.
The tricky part is keeping all of that consistent. If a message in the simulation says a number dropped, the dataset has to show exactly that drop. So each simulation has a script that loads the data the same way the candidate’s browser does and checks every number the scenario mentions.
We test the grader in a similar way. After any change, we run a strong and a weak synthetic candidate through the whole pipeline and check that the report still tells them apart.
Defending the work
At the end of a simulation, the candidate is asked to walk through what they did and defend it. The questions are generated from their own work, and the model writing them never sees the expected answers.
The same signals can be evaluated for a fresh graduate or for someone with years of experience. The environment stays the same and the rubric changes.
What we learned
The old formats weren’t a mistake. They were the best anyone could do when a computer could only check whether two things were equal. Many of the formats we now think of as an assessment came from that limit.
Taking the limit away didn’t make evaluation easy, though. The problem moved from writing questions a computer can mark to making sure an AI’s judgement can be checked. That is why every score on the report has to point at something the candidate actually did.
What matters most is what this does to scale. Testing thousands of candidates used to force employers to turn the job into aptitude questions. That is no longer true, so the test that thousands of candidates take can now look much more like the work itself.