Students Will Game Your Grader: Trusting Test Output Across Five Toolchains

"Tests run: 11, Failures: 6." A few lines later: "BUILD SUCCESS." Exit code 0.

That's what Maven printed when I set one property in the pom.xml of a Java repo with every planted bug still in place. My grader decided pass or fail from the exit code. It would have approved the PR.

Kadra is a simulated software job. You get a real public GitHub repo with planted bugs, AI teammates, and a ticket a day. Each PR is graded by hidden tests running in a fresh Vercel Sandbox microVM. There are eight stacks across Node, Python, Go, Java/Spring and C#/.NET, so five test runners with five ideas of what success looks like.

The uncomfortable fact underneath it all: the student owns every file in the repo, so they own every file the test command reads. This is how I stopped trusting that command, including where my fixes were wrong first.

The grader believed a number the student controlled

The original grader took its verdict from one signal: the exit code of the last test step. Nothing was restored from a canonical source except the grading tests themselves.

While reviewing the PR that shipped the Java pack, I reproduced two bypasses against a fully defective tree:

• Set maven.test.failure.ignore to true in pom.xml. Six tests fail, Maven reports BUILD SUCCESS, exit 0.
• Add .mvn/maven.config containing -Dmaven.test.skip=true. No tests run at all. Exit 0.

Java was just the first pack where one line did it. Node's vitest.config.ts and package.json, Python's conftest.py and pytest.ini, Go's go.mod: none of them were restored before grading.

The same trust caused a quieter bug in the other direction. A compile failure, a missing toolchain or a wrong JAVA_HOME exits non-zero, and the grader recorded it as the student's failed fix.

The fix has two halves, and the design spec is blunt that neither works alone. Restoring config removes the student's ability to neutralise the suite. Parsing output removes the grader's willingness to read silence as success. Parse without restoring and the student forges the line the parser wants. Restore without parsing and a build that fails to compile reads as a caught defect.

A verdict from evidence, in five dialects

The verdict now comes from a pure module, src/domain/test-summary.ts, that parses vitest, pytest, go test -v, Maven Surefire and dotnet test output into two numbers: tests that ran, and tests that failed. Then:

• Tests ran, none failed, exit 0: passed.
• Tests ran, none failed, non-zero exit: failed. The text and the status disagree.
• Any failure: failed.
• Nothing ran: failed. The suite selected nothing.
• No summary at all: an infrastructure failure, reported separately so nobody is blamed for a broken toolchain.

Three rules make the parsing defensible.

Rule 1: anchor on the real summary line

The code under test runs inside the test process, so anything it prints lands in the same stream. A bare search for "6 passed" will happily read a console.log planted in the student's source. Every pattern matches the runner's real summary shape at the start of a line. Two of them, simplified for this post:

// Maven prefixes its own lines; a test's System.out is printed bare.
const SUREFIRE_TOTALS =
  /^\[(?:INFO|ERROR)\][ \t]+Tests run: (\d+), Failures: (\d+), Errors: (\d+)/gm;
// pytest: counts at line start, then the session duration ("in 0.03s").
const PYTEST_SUMMARY =
  /^[ \t]{0,}(?:=+[ \t]{0,})?((?:\d+ [a-z]+)(?:[ \t]{0,},[ \t]{0,}\d+ [a-z]+){0,})[ \t]+in[ \t]+[\d.]+s/gm;

None of this is something to guess at, so the parsers were written against sixteen fixtures captured from real sandbox runs (pass, fail, empty and broken for each of the first four toolchains), with four more when C# arrived.

They earned their keep. An empty pytest run doesn't print "0 passed"; it prints "17 deselected in 0.02s". A C# run where every test is skipped prints "Total: 1" and exits 0, so the .NET parser counts Passed plus Failed and never reads Total. vitest's "Tests" line counts only executed tests, so a grading file that throws on import contributes zero and the line stays green; the parser reads the "Test Files" line too.

Rule 2: read every summary line and take the worst

I got this wrong twice before merging.

My first vitest and pytest parsers took the first match. Review fed them adversarial output and four cases graded green, including fake vitest and pytest summaries printed by the code under test. Runners print their summary last, so I switched to the last match.

The next review round broke that too. Python's atexit hooks fire at interpreter shutdown, after pytest has printed its summary, and the grading tests have to import the student's module. One hook printing "2 passed in 0.02s" landed after pytest's real "1 failed, 1 passed" line. Last-match read a fully defective tree as a clean pass.

Position isn't a defensible rule in a stream the student can write to. What student code can't do is take back a line the runner already printed. So every summary-shaped line is read and the worst of each count wins (simplified):

function worst(summaries: TestSummary[]): TestSummary {
  return {
    ran: Math.max(0, ...summaries.map((s) => s.ran)),
    failed: Math.max(0, ...summaries.map((s) => s.failed)),
  };
}
// exitCode is required, not optional: an optional security check is one a caller forgets.
export function verdictFrom(summary: TestSummary | null, exitCode: number): GradeVerdict {
  if (!summary) return { kind: "infrastructure", reason: "no test summary found" };
  if (summary.ran === 0) return { kind: "failed", reason: "no tests ran" };
  if (summary.failed > 0) return { kind: "failed", reason: "grading tests failed" };
  if (exitCode !== 0) return { kind: "failed", reason: "summary and exit code disagree" };
  return { kind: "passed" };
}

Rule 3: the exit code is necessary, never sufficient

Trusting the exit code alone was the original bug. Ignoring it throws away the one signal printed text can't forge. A green summary from a command that exited non-zero is a contradiction, and it never passes.

Restore before install, then burn the answer key

The other half is making sure the config the runner reads is mine. Before install, the grader shallow-clones the pack's template repo and runs one set -e script: delete denied paths, delete denied file names at any depth, copy protected paths over the student's tree, delete the clone. The real generated script for a Python pack:

set -e
rm -rf 'pyproject.toml' 'tox.ini' 'setup.cfg'
find . -name 'conftest.py' -not -path './.git/*' -delete
rm -rf 'requirements.txt'
mkdir -p "$(dirname 'requirements.txt')"
cp -R '/tmp/canon/requirements.txt' 'requirements.txt'
rm -rf 'pytest.ini'
mkdir -p "$(dirname 'pytest.ini')"
cp -R '/tmp/canon/pytest.ini' 'pytest.ini'
rm -rf '/tmp/canon'
# only now: pip install, wipe tests/grading, write the canonical tests, run pytest

Before install, not before test. npm ci and pip install execute code the manifest names: a postinstall hook, or a fake package installed under the test runner's name that prints a flawless summary. Restoring package.json afterwards doesn't uninstall what already ran.

Deny by name, at any depth. pytest collects conftest.py from the root directory down to the test directory, so the root, tests/ and tests/grading/ are three separate places to install a hook that reports every test as passed. A path-based deny closes one of them.

Delete the clone before any student code runs. The template is the un-bugged source; defects are planted by patch after the student's repo is generated. Left on disk, one copy command from inside a test run makes every grading test pass for real.

Keep the token off disk. The GitHub App installation token can write to every sprint and template repo. It was originally in the clone URL, which left it in /tmp/canon/.git/config for the whole student-controlled run. It's now a per-command environment variable read by an inline credential helper, passed with git -c before the subcommand, because git clone -c writes the value into the new repo's config.

After install, the grading test directory is emptied and refilled with the canonical tests alone. Anything a student adds there, like an extra green vitest file masking a canonical one that died on import, isn't config, so restore can't reach it.

The guard tests caught me

Two entries in my first draft of those lists would have broken production. I had pytest.ini on the deny list, but both Python templates ship one carrying pythonpath = ., the only reason the package imports resolve. Deleting it would have failed every legitimate Python submission. And I had go.sum on the Go protected list, but the Go template has no external dependencies and no go.sum, so the copy would have failed every Go grade.

Guard tests asserting that every protected path exists in the template, and no denied path does, caught both before merge. A missing protected path is an error, not a skip: the list has drifted and the protection is imaginary.

C#, where an added file is more powerful

.NET compiles every source file under a project directory, and a [ModuleInitializer] method runs before any test does. So the grading tests live in their own project, and the grader runs dotnet test against that project file alone. It compiles the grading project and its reference to the app, never the student's own test project, so an initializer planted there never runs.

My design doc called that a structural guarantee. Final review proved it wasn't. The SDK's default compile glob starts at the project directory, not at the Cases folder the grader wipes. A file dropped elsewhere in the grading project, declaring a correct BookingService in the same namespace, shadows the real one, because C# prefers a source-declared type over a referenced one and only warns. The canonical suite passes against the shadow with the planted bug untouched. The fix lives in the protected project file: default compile items off, compilation pinned to the one folder the grader refills.

MSBuild also earned the longest deny list of any stack. Directory.Build.props is auto-imported from any ancestor directory and can run arbitrary build tasks. nuget.config is denied in both casings, because NuGet matches the name case-insensitively and Linux doesn't.

Proving a bypass is actually blocked

scripts/verify-pack-grading.mjs runs the grader's production shell strings in a real sandbox, imported from the same module rather than copied; the only step it substitutes is the private template clone. For each ticket, the grading test must fail on the buggy tree and pass on the fixed one. Then every known bypass is applied to the buggy tree, run through the full pipeline, and must not pass.

That check has a hole of its own: "the run didn't pass" is equally true when a bypass is defended and when it just broke the tree. Two of my pytest vectors did exactly that. They injected code above from __future__ import annotations, the module failed to load, pytest reported one error, and both checks went green without the fake line ever printing. Now every bypass must reproduce the buggy tree's own signature: the same failed count, and at least as many tests run. After the fix, the atexit vector on the fintech pack reads 17 ran, 3 failed. The forgery inflated the ran count; the three real failures still sank it.

The latest sweep across all eight packs came back seven clean: 21 tickets, 21 independence checks, 43 bypass vectors rejected. The eighth hit the failure mode the spec calls the dangerous one, rejecting correct work. An e-commerce grading test used a percent coupon, the exact branch another ticket's planted bug breaks, so a correct fix graded red. A fixed coupon made the test measure only its own ticket.

The other ways it lied

Production was grading with a mock. In late July I found the real grader was only selected when VERCEL_OIDC_TOKEN was set. It isn't set at Vercel function runtime, where the token arrives as a request header, and grading runs in a detached waitUntil task. So production silently fell back to MockGrader, whose default result is a pass. The grader now uses explicit Sandbox credentials, and a deployment with none throws instead of falling back.

The advisory check lied green. Every template ships a grade.yml workflow so students get an early signal in their own PR. It restored protected paths but ignored the deny lists, so a student who added a conftest.py saw a green check, then a failing grade. The real grade was never fooled, but the check taught the wrong lesson. Its deny step is now generated from the grader's own function, and a test asserts each workflow contains that exact output.

An SDK default flipped. Between @vercel/sandbox 2.3.0 and 3.0.0, persistent changed its default from false to true. Without the explicit persistent: false already in the code, every grading run would have been snapshotted into the 15 GB lifetime storage quota.

What this doesn't claim

It doesn't make cheating impossible. A student who rewrites their source so the canonical tests genuinely pass has done the ticket. This closes neutralising the suite, not writing code that passes it.

It also isn't airtight. The spec says so explicitly and lists known gaps I haven't closed. What the rules buy is that a bypass has to beat all of them at once, which no single planted line does.

Lessons learned

1. An exit code is a claim, not evidence. Skipped suites, ignored failures and green runs all exit 0. Decide from parsed output and keep the exit code as a veto.

2. Never choose by position in a stream the adversary can write to. First match lost to a forgery printed during the run, last match to one printed after it. Worst-line-wins held both ways.

3. Ordering is a security property. Restore before install, because install already runs student code. Delete the answer key before the first student process starts.

4. Make sure your bypass tests fail for the right reason. A vector that breaks the tree scores green without testing anything.

5. Pin security lists to the source of truth. Checking protected and denied paths against the template caught two mistakes that would have failed every Python and every Go submission.

6. A mock in production is a silent pass. If the real dependency is missing on a deployment, throw. Visibly broken grading is recoverable; quietly approving everything isn't.

Building something that has to trust the output of code you didn't write? Let's talk, or see how it fits together in Kadra.