GitHub said it replaced the large language model that powers its secret-scanning push protection with a two-stage pipeline built around a smaller, fine-tuned classifier, according to a post on the GitHub Blog.
The new system first uses regex matching to flag credential-like patterns, then passes candidates to a fine-tuned ModernBERT classifier that reads the surrounding code, file paths and variable names to decide whether a match is a real secret or a false positive such as a placeholder or test fixture, GitHub said.
In A/B testing against the prior LLM-based pipeline, the new model raised repository owner-weighted acceptance of its findings by 11.5% and cut median scan time after a push by 86%, GitHub said. The company said serving costs fell 98% compared with calling a remote LLM, and offline evaluation showed 13% higher precision and 19% higher recall.
The classifier exports to the ONNX format with 8-bit quantization, which GitHub said keeps inference fast enough to run on every push without the latency of a large language model call.
GitHub is describing a tradeoff that cuts against the industry's usual direction: a smaller, purpose-built model outperforming a general LLM on a narrow task, at a fraction of the cost. Teams running their own classification pipelines on top of LLM calls have a concrete data point for when a fine-tuned small model is worth building instead.