TL;DR
Get privacy and security gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Google says Gemini 4 Carbon scored highly on three cybersecurity benchmarks for finding, exploiting and fixing software vulnerabilities. Two tests were run by Google, and comparisons with rival models use results from different reports and methods; no independent Carbon evaluation is public.
Google says its Gemini 4 Carbon model scored 88.2% on a vulnerability-finding benchmark, 54.8% on an exploit benchmark and 71.0% on a software-fix benchmark. The scores point to capabilities that could help security teams, but two tests were run by Google, published comparisons use different methods, and the company says access is currently limited to vetted cyber defenders.
The three tests cover different tasks. SEC-bench Pro asks a model to produce an input that triggers a known software bug after giving it the relevant code area, bug type and expected error. Google reports Carbon’s 88.2% result on a 183-task version covering the V8 and SpiderMonkey browser engines. The test is based on bugs that had already been fixed; it does not measure finding an unknown vulnerability from scratch. Google used its Antigravity agent tool and averaged three attempts.
ExploitGym tests whether a model can turn a crash into an exploit that runs code and retrieves a hidden flag. Google reports a 54.8% score with a 10-hour limit. Its methodology note also gives a 53.5% result within six hours. The test was run with standard protections such as memory randomization disabled, according to the source material. Google ran this benchmark as well.
CWE-bench asks a model to audit an open-source code base and fix its flaws without being told where they are. A patch counts only if it blocks the vulnerability and existing tests still pass. Collinear AI, which created the benchmark, ran Carbon on a private set of 120 tasks and reported a 71% score. The source says five tasks defeated every tested model and that Carbon and several rivals scored 68%. These results do not establish how the models would perform on every real-world code base.
What the Scores Mean for Defenders
The results matter because the same model may assist with finding weaknesses, demonstrating their impact and proposing repairs. That could help defenders triage security work, but the exploit results also show why access controls matter: a capability useful for testing can also be misused. Google says Carbon is available first to a small group of vetted cyber defenders, rather than generally to users.
The fixing result also leaves room for human review. A 71% benchmark score means the model did not pass roughly three in ten tasks on its first try, according to the source material. Security teams considering AI-generated patches would still need to check that a repair closes the flaw without causing other problems. Benchmark performance is evidence about performance on those tests, not a guarantee of safe or correct changes in production systems.
Products Worth Considering
Logitech MX Mechanical Full-Size Wireless Mechanical Keyboard, Tactile Quiet Switches, Backlit Keys, Bluetooth, USB-C, Multi-Device, Graphit
Trezor Safe 5 Crypto Hardware Wallet with Color Touchscreen, Black Graphite
Ledger Flex + Folio Crypto Hardware Wallet
As an affiliate, we earn on qualifying purchases.
How the Three Tests Differ
The benchmarks represent separate stages of security work. SEC-bench Pro gives models a constrained search area and asks them to trigger a known bug. ExploitGym begins with a crashing input and measures whether the model can turn it into a working attack. CWE-bench gives models a code base and asks them to identify and repair issues without specifying their location.
The source material reports that Google produced Carbon’s SEC-bench Pro and ExploitGym results, while Collinear AI ran CWE-bench on its private test set. It also cites OpenAI’s 85.4% SEC-bench Pro result for GPT-6 Astra on the same version and Gemini 4 Argon’s 83.4%. For CWE-bench, Carbon’s reported 71% was three points above the listed 68% scores for GPT-6 Astra, Gemini 4 Argon and Grok 4.7, and four points above Claude Opus 5.5 at 67%.
ExploitGym comparisons require particular care. Google’s 10-hour Carbon result is not directly comparable with Anthropic’s reported 34.5% for Claude Opus 5.5 under a six-hour limit. The figures also come from separate lab reports. The source says the labs tested with safety filters off, another methodological detail that affects how to interpret the numbers.
“Gemini 4 Carbon scored 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench.”
Where the Evidence Is Limited
No independent public test of Carbon is cited in the source material. Two of the three headline scores come from Google-run evaluations, while the CWE-bench result comes from the benchmark’s creator on a private test set. The private tasks may reduce the risk that models have seen the answers during training, but they also limit outside scrutiny of the evaluation.
It is also unclear how directly the scores predict results on live systems, where code, access controls and defenses vary. The reported comparisons do not all use matching time limits, graders, tools or reporting methods. Google says Carbon is in the U.S. government’s voluntary pre-release testing, but the source does not provide findings from that process or a public timetable for release.
Access and Testing Ahead
Google says it is providing Carbon to a small group of vetted cyber defenders and that the model is part of the U.S. government’s voluntary pre-release testing. The source material does not state when broader access might begin or when results from government testing will be published.
Further independent evaluations could help clarify how Carbon performs across different tools, safeguards and real software. Until then, the reported scores describe results on specific benchmarks. For companies, they offer a reason to examine methods and plan for human review of AI-generated fixes; for ordinary users, the source says there is no immediate change, and keeping software updated remains the practical protection.
Key Questions
What did Google report about Gemini 4 Carbon’s security scores?
Google reported 88.2% on SEC-bench Pro, 54.8% on ExploitGym with a 10-hour limit, and 71.0% on CWE-bench. Each test measures a different security task.
Has an independent group tested Carbon?
The source material says no independent public evaluation of Carbon’s security skills is available. Google ran SEC-bench Pro and ExploitGym; Collinear AI ran CWE-bench.
Can the ExploitGym score be compared directly with rival models?
Not on the information provided. Google’s headline Carbon score uses a 10-hour limit, while the source cites a six-hour result for Claude Opus 5.5. Results also come from separate reports and may use different methods.
Who can use Gemini 4 Carbon now?
Google says access is limited to a small group of vetted cyber defenders. It has not provided a general release date in the source material.
Do these scores show that Carbon can reliably fix software vulnerabilities?
No. The 71% CWE-bench score is a result on a particular private set of tasks, and the source says the best model still failed roughly three in ten on the first try. It does not establish reliable performance on every code base.
Source: Thorsten Meyer AI
