How evaluation works
What CodeXams checks with code, what a model only proposes, what a person decides, and why a result never has a single score.
From the exam to a signed result
The exam
The candidate solves one task in a timed exam. The editor records every edit, paste and run; with the assessment's policy, the screen too.
Hidden tests, by tier
Checked by codeWhen the program is submitted, code runs hidden tests in three tiers and counts what passed in each.
Questions about their own code
Checked by codeShort questions about the candidate's own program. Code checks the answers to predict, find the bug and change the program; explanations wait for a person.
A model's proposal
Proposed by a modelA model proposes a score per rubric dimension, each citing evidence that exists, and a second model checks it. Labelled "AI-proposed, not validated".
The routing policy
Checked by codeFixed rules read the recorded facts and send the exam to a full review, to a check of a drafted decision, or mark every check passed. It drafts a decision; it decides nothing.
A person decides
Decided by a personA reviewer reads the proposal first, keeps or changes each rubric score (a reason for a change is optional), and records the decision.
An owner validates and publishes
Decided by a personAn organisation owner confirms every part (the answers code checked, the explain ratings, the rubric scores, the band, the integrity observations, the decision), then publishes. Until then the candidate sees only that the exam is being reviewed.
The signed result
Checked by codePublishing signs the result, so anyone can check it was not changed and whether it was revoked. Each part then reads "Validated by" the organisation.
Hidden tests, in three tiers
The hidden tests run on the submitted program and are reported per tier, never added up.
Small tier
Small inputs, including the edge cases: is the answer right?
Medium tier
Bigger inputs: a correct but slow solution still passes.
Large tier
Large inputs: only an efficient solution passes in time.
Questions about the candidate's own code
They carry the label “Not yet validated”: they have not been validated as a measure, so no decision rests on them alone.
Predict the output
Checked by codeWhat their program prints for a given input. Code compares the answer with a run.
Find the bug
Checked by codeA one-line change is made to their program so it fails a test: name the changed line and an input that exposes it. Code checks both parts.
Change the program
Checked by codeA changed requirement: change their program to meet it. Code runs the changed program against new hidden tests.
Explain the code
Decided by a personExplain a part of their program in words. A reviewer rates it 0, 1 or 2 against the question's rubric; two models may propose a rating.
The rubric a reviewer scores
A reviewer scores five dimensions from 1 to 5, from the evidence, before seeing a model's proposal. The reviewer's decision is the one that counts.
- Problem understanding: Did they read and frame the problem before coding?
- Use of AI: Guided use (asks for help on parts, adapts the output) versus delegation (pastes the task, pastes the answer unchanged).
- Verification: Ran tests, handled errors, checked edge cases after adopting code.
- Understanding of their own code: From the questions about the code: predict, find the bug, modify, explain.
- Process: Exploration, recovery from failures, iteration.
Why there is no total score
Small tier
11 of 11
Large tier
0 of 2
Predict the output
3 of 3
What a result shows, and what it does not
What a result shows
- Whether the submitted program passes hidden tests, per tier.
- Whether the candidate can answer questions about their own program, per question type.
- How the work was done: runs, pastes and where they came from, pauses in screen sharing.
- A reviewer's judgement on five rubric dimensions, starting from the model's proposal, and their decision.
What it does not
- Skill in general: a result records one exam and one decision by one organisation.
- Anything from the CV, a name or the organisation: the model sees only the exam's evidence.
- A rank or a comparison of candidates.
- Misconduct: integrity observations are observations for a person to read, never findings.