When we released 10 billion tokens of synthetic code, the most common question wasn’t about generation — it was about validation. Generating plausible code is cheap. Keeping only the code that’s actually correct is where the work is.
Every sample in the release passed a four-stage gauntlet: it has to parse, compile, execute against generated tests, and survive a mutation check that catches tests that pass vacuously. Roughly 40% of raw generations fail somewhere in that pipeline, and that rejected 40% is exactly the data you don’t want in a training run.
The pipeline itself is boring on purpose. Each stage is a containerized pass/fail check with no model in the loop, so results are reproducible and the whole thing parallelizes across cheap hardware. The expensive models generate; the cheap infrastructure filters.
We also publish per-language quality breakdowns with the dataset, because aggregate acceptance rates hide a lot. Rust samples fail compilation more often but survive mutation testing at higher rates; the inverse holds for Python. If you’re filtering the dataset for your own run, those slices matter.
The validation harness ships alongside the data. If you only take one thing from this post: publish your filters, not just your tokens. A dataset is only as trustworthy as the pipeline that produced it.
