Governance
Configure fix approvals, protected paths, and account settings in one place.
Security commentary
Why Defenders Can’t Bet on One Model
Design model selection around measured tasks, reliable tools and evidence that survives a provider change.
The workload is a collection of tasks
A security investigation may require locating code, explaining a data flow, generating a test fixture and interpreting an execution result. These tasks have different context, latency and verification needs. Selecting a model using one broad preference can hide those differences. A useful architecture begins by identifying which decisions require model reasoning and which can be handled by a deterministic tool or a direct check.
This does not imply that every deployment needs a complex model router. A single model can be a sensible starting point. The important property is that its performance and operating limits are measured against the actual workload, rather than assumed from an unrelated leaderboard or a memorable demonstration.
Compare models under the same conditions
Build an evaluation set that reflects the repositories, languages and issue types the workflow will encounter. Freeze the tool definitions, retrieved context, time allowance and acceptance criteria while comparing models. Include clean examples and incomplete cases as well as confirmed vulnerabilities. Track whether the system reports uncertainty accurately when it cannot establish a conclusion.
Measure the total cost of reaching a useful decision. A cheaper call can become expensive if it produces many invalid tests, repeated retries or findings that require lengthy human investigation. A slower model can also be unsuitable when a bounded, faster analysis provides enough evidence for the task. Retain traces so the tradeoff can be inspected rather than reduced to one aggregate score.
Keep the security boundary independent
A fallback model should inherit the same account scope, tool permissions and data restrictions as the primary model. Provider availability should not determine whether a workflow can access a production secret or reach a new network destination. If a task cannot run safely with the available capabilities, return a clear incomplete result. Do not turn a failed request into permission for a broader execution path.
Normalise the evidence you require from each model: source locations, explicit assumptions, proposed tests and results from actual tools. Separate those fields from free-form explanations. This gives reviewers a consistent way to assess a finding even when the underlying model changes, and makes unsupported claims easier to identify.
Treat upgrades as observable changes
Record the model identifier and relevant configuration with each evaluation and production run. Before an upgrade, replay representative cases and compare the quality of the resulting decisions. Include difficult negative cases, tool failures and manipulated retrieved content. A new model can improve one part of the workflow while changing behaviour elsewhere. Retaining the previous configuration makes a controlled rollback possible if the new behaviour causes problems.
Use disagreements as a reason to inspect evidence, not as an automatic vote. Two models can repeat the same mistaken assumption if they share the same incomplete context. Independent verification of the security property is more valuable than consensus over prose. Defenders benefit from optionality when it is supported by clear interfaces, measured performance and an execution environment whose boundaries do not depend on which model is currently available.
