Skip to content

How evaluation works

What CodeXams checks with code, what a model only proposes, what a person decides, and why a result never has a single score.

From the exam to a signed result

From the exam to a result anyone can check. After the exam, each step is done by code, proposed by a model or decided by a person, and says so wherever it appears.
  1. The exam

    The candidate solves one task in a timed exam. The editor records every edit, paste and run; with the assessment's policy, the screen too.

  2. Hidden tests, by tier

    Checked by code

    When the program is submitted, code runs hidden tests in three tiers and counts what passed in each.

  3. Questions about their own code

    Checked by code

    Short questions about the candidate's own program. Code checks the answers to predict, find the bug and change the program; explanations wait for a person.

  4. A model's proposal

    Proposed by a model

    A model proposes a score per rubric dimension, each citing evidence that exists, and a second model checks it. Labelled "AI-proposed, not validated".

  5. The routing policy

    Checked by code

    Fixed rules read the recorded facts and send the exam to a full review, to a check of a drafted decision, or mark every check passed. It drafts a decision; it decides nothing.

  6. A person decides

    Decided by a person

    A reviewer reads the proposal first, keeps or changes each rubric score (a reason for a change is optional), and records the decision.

  7. An owner validates and publishes

    Decided by a person

    An organisation owner confirms every part (the answers code checked, the explain ratings, the rubric scores, the band, the integrity observations, the decision), then publishes. Until then the candidate sees only that the exam is being reviewed.

  8. The signed result

    Checked by code

    Publishing signs the result, so anyone can check it was not changed and whether it was revoked. Each part then reads "Validated by" the organisation.

Hidden tests, in three tiers

The hidden tests run on the submitted program and are reported per tier, never added up.

Three tiers of hidden tests, by input size
  1. Small tier

    Small inputs, including the edge cases: is the answer right?

  2. Medium tier

    Bigger inputs: a correct but slow solution still passes.

  3. Large tier

    Large inputs: only an efficient solution passes in time.

Questions about the candidate's own code

They carry the label “Not yet validated”: they have not been validated as a measure, so no decision rests on them alone.

  • Predict the output

    Checked by code

    What their program prints for a given input. Code compares the answer with a run.

  • Find the bug

    Checked by code

    A one-line change is made to their program so it fails a test: name the changed line and an input that exposes it. Code checks both parts.

  • Change the program

    Checked by code

    A changed requirement: change their program to meet it. Code runs the changed program against new hidden tests.

  • Explain the code

    Decided by a person

    Explain a part of their program in words. A reviewer rates it 0, 1 or 2 against the question's rubric; two models may propose a rating.

The rubric a reviewer scores

A reviewer scores five dimensions from 1 to 5, from the evidence, before seeing a model's proposal. The reviewer's decision is the one that counts.

  • Problem understanding: Did they read and frame the problem before coding?
  • Use of AI: Guided use (asks for help on parts, adapts the output) versus delegation (pastes the task, pastes the answer unchanged).
  • Verification: Ran tests, handled errors, checked edge cases after adopting code.
  • Understanding of their own code: From the questions about the code: predict, find the bug, modify, explain.
  • Process: Exploration, recovery from failures, iteration.

Why there is no total score

One example exam. Every small test passed and no large one did: the program is right but too slow. Added up, the 14 of 16 would read as “nearly all” and hide exactly that. So each part keeps its own scale, and a person reads them together.
  • Small tier

    11 of 11

  • Large tier

    0 of 2

  • Predict the output

    3 of 3

What a result shows, and what it does not

What a result shows

  • Whether the submitted program passes hidden tests, per tier.
  • Whether the candidate can answer questions about their own program, per question type.
  • How the work was done: runs, pastes and where they came from, pauses in screen sharing.
  • A reviewer's judgement on five rubric dimensions, starting from the model's proposal, and their decision.

What it does not

  • Skill in general: a result records one exam and one decision by one organisation.
  • Anything from the CV, a name or the organisation: the model sees only the exam's evidence.
  • A rank or a comparison of candidates.
  • Misconduct: integrity observations are observations for a person to read, never findings.