Governance
Configure fix approvals, protected paths, and account settings in one place.
Research
Security evaluation benchmark
Explore the trade-off between security-task recall and evaluation cost, compare sample pipelines, or load your own local CSV results.
Illustrative sample data — not measured product results.
Sample values demonstrate cost–recall trade-offs. Upload a local CSV with name, objective, cost and recall columns to explore your own results. The file is not uploaded.
Compare complete systems, not just model names
A security evaluation should measure the whole pipeline: the model, tools, repository context, task budget, validation process and reporting rules. Changing any of those components can change the outcome. A score without those conditions is difficult to reproduce or interpret.
The interactive explorer above starts with illustrative sample pipelines. These numbers are examples for learning how the chart and filters work. They are not measured Counterswarm product results, rankings of commercial models, or a claim of research ownership. Load your own CSV to compare actual runs under a documented evaluation protocol.
Choose an objective before interpreting recall
Detection asks whether a system identifies a relevant weakness. Validation asks whether the reported weakness is supported by evidence of a real security impact. Differential analysis asks whether a particular change introduced or removed a weakness. These are related tasks, but success on one does not establish success on the others.
Use a consistent definition of a correct result. Record how ambiguous cases, duplicate findings, partially correct explanations and failed tool calls are handled. A human review process should be applied consistently across all compared pipelines rather than only to promising outputs.
Read the cost–coverage frontier
A pipeline is dominated when another pipeline achieves at least as much recall at no greater cost, with an improvement on at least one dimension. Showing the nondominated frontier makes the trade-off easier to inspect. It does not identify one universally best pipeline: a team may prefer a cheaper option for routine review and a more expensive option for a narrow, high-impact assessment.
Treat cost as an explicitly defined quantity. Model-token charges, compute, external tool fees, analyst time and repeated attempts are different expenses. The explorer uses the supplied cost per evaluated task in US dollars; make sure all uploaded rows use the same cost definition.
Make an evaluation reproducible
Keep the task set separate from prompt development and record the exact revision of code or configuration being assessed. Document tools, network access, time limits, retry rules, context selection and output review. A held-out test set reduces accidental tuning to the answers, but it does not remove every source of data contamination or selection bias.
Run enough independent tasks to understand variation. If outcomes are binary, report uncertainty around the success rate and inspect failure categories. A small difference in a single run is weak evidence for a durable capability advantage. Compare on representative workloads and repeat after material system changes.
Use your own local results
CSV files should contain name, objective, cost and recall columns. The objective must be detection, validation or differential; recall is a percentage from zero to one hundred. The file is read in your browser and is not uploaded by this explorer. Export downloads the currently filtered rows so that your comparison can be reviewed with its evaluation notes.
Before publishing a real benchmark, retain the task definitions, scoring rubric and provenance for the results. Explain what was measured and what was not. Benchmark performance is evidence about a defined evaluation, not a general guarantee that a system will prevent every attack.
