GitHub’s smaller secret scanner: when AI should classify, not chat

Imagine a password-shaped string in a repository. It might be a live database credential, a test fixture or a placeholder in a setup guide. A regular expression can flag the shape, but it cannot reliably tell what the string means. That ambiguity creates two expensive failures: missed leaks and alerts that developers learn to ignore.

Two stages, two different jobs

GitHub’s first stage uses regex filters to find likely generic secrets quickly. It casts a wide net. The second stage sends each candidate, together with context such as its file path, variable names and nearby code, to a fine-tuned ModernBERT classifier. The classifier scores whether the match is a real credential or a lookalike. High-confidence findings become alerts; likely examples, checksums and test values can be filtered out.

This is classification, not a chatbot answering a security question. The model returns a decision rather than generating an explanation or prose for every candidate. GitHub says that makes it faster and cheaper to serve at its scale. There is still a hard boundary: a real secret the regex stage never proposes cannot be rescued by the classifier.

The design also changes when a warning can appear. GitHub says the faster pipeline made generic-secret detection practical in push protection, where a developer can be stopped before a credential enters a repository. Post-push scanning still creates alerts for material already committed.

What GitHub actually measured

Against its previous LLM-based pipeline, GitHub reports 13% higher precision and 19% higher recall in an offline evaluation using the same public post-push traffic. A customer A/B experiment found 11.5% higher repository owner-weighted alert acceptance. GitHub also reports 86% lower median post-push scan execution time and more than 98% lower model-serving cost compared with remote LLM calls.

Those numbers describe GitHub’s workload, evaluation and infrastructure. The serving-cost figure is not a reduction in every cost of running secret protection, and it is not a saving another team should budget for sight unseen. Acceptance is a useful signal of alert quality, but it is not the same measurement as ground-truth precision.

The useful engineering lesson

If you are building a detector, start by separating candidate discovery from the ambiguous judgment. Measure the first stage’s recall: which known secrets never reach the model? Then collect hard negatives that look like secrets but are not: documentation placeholders, generated IDs, hashes and test fixtures. Test on held-out repositories and values, not near-duplicates of training examples. Track precision and recall together, because an apparently quieter scanner can simply be missing more leaks.

GitHub describes combining human-validated examples, weakly supervised labels and generated edge cases, then exporting the trained model to ONNX with 8-bit quantization for serving. The transferable skills are less glamorous than choosing the largest model: dataset design, leakage-resistant evaluation, latency measurement and fitting inference to the actual production path.

The takeaway is that generative AI is not the default tool for every task involving code. When the output is a repeated, testable yes-or-no judgment, a specialist classifier may be a better fit. GitHub’s result is a strong production example, not a universal benchmark or a downloadable GitHub model you can simply drop into your own scanner.