If a single AI review comment has started stopping merges
At first, AI code review feels reassuring. It catches internal rules people miss and even links to the supporting documentation. But the moment those comments are wired into required checks, a judgment that misreads the context can block a real merge. Developers may start looking for ways around the check instead of following the rule.
Cloudflare did not solve this problem through model performance alone. It manages engineering standards as RFCs and separates a document being approved from being ready for automated blocking. Approved rules first appear as non-blocking recommendations; only unmet MUST requirements in RFCs that have separately entered an enforcement state are used to hold approval or block merges.
The takeaway is not a giant AI review system. It is an operating model: every rule needs a requirement level, stable ID, scope, source text, and enforcement state so AI feedback can be tracked and challenged.
Before citing 16,000, read the measurement scope
In the opening of a post published in August 2026, Cloudflare said its AI code reviewer had flagged nearly 250,000 standards deviations and prevented 16,000 merges over the preceding four months. In the detailed section, though, it says that since Codex was introduced, it had flagged roughly 230,000 issues and held approval on around 16,000 changes. Since the start dates and inclusion criteria for these counts have not been disclosed, they should not be combined into a single precise performance figure.
Detection volume is not the number of quality improvements.
The total or unique number of merge requests, repeat judgments on the same change, false-positive rate, and developer acceptance rate for fixes have not been disclosed. This is less evidence of AI review accuracy than an operational example of the enforcement structure needed when many judgments are connected to the development flow.
Specification-review metrics need the same reading. From early May 2026 through publication, Cloudflare's AI spec reviewer reviewed about 600 unique public specs and ran more than 3,200 times, including reruns after changes. The severity mix of findings was 65% major, 29% minor, and 6% critical, but the total number of findings was not disclosed, and these percentages do not indicate accuracy or the actual defect-prevention rate.
Approval and enforcement are different switches
When an RFC in Cloudflare Codex becomes approved, the AI reviewer can report related violations, but the result is non-blocking. Only after the RFC is promoted in a separate step to enforced can unmet MUST requirements lead to held approval or blocking according to severity. Cloudflare has not disclosed who approves promotion or what false-positive rate or observation period is required.
| Rule state | AI review behavior | What the team must decide separately |
|---|---|---|
| approved | Non-blocking recommendation with supporting evidence | Scope and recurring exceptions |
| enforced | Use unmet MUST requirements to hold approval or block | Promotion authority, appeal path, and unblock path |
MUST and SHOULD are not equivalent commands. In RFC 2119, MUST is an absolute requirement, while SHOULD permits valid reasons to depart in particular circumstances, provided the implications are fully understood and carefully weighed. Strong requirement language should be used carefully when it is genuinely needed for interoperability or to prevent harm. Turning every best practice into a MUST does not strengthen enforcement; it may simply create more reasons to block.
Google's public code-review guidance also prioritizes continuous improvement of the codebase over perfection and recommends marking unimportant feedback as optional. The AI review interface should likewise clearly distinguish required changes from recommendations, so every comment does not become a de facto blocking order.
A stable rule ID matters more than a long rulebook
Cloudflare has more than 60 RFCs, so it does not put every full document into the model each time. A separate agent extracts MUST and SHOULD statements into JSON and attaches the RFC number, title, state, domain, section, requirement level, and source link. The reviewer searches this compressed index first and loads the full RFC only when it needs more context.
Each requirement statement has a stable slug that remains intact after an RFC is revised. That lets multiple systems continue tracking judgments, analysis, and exceptions for the same rule. Cloudflare's internal AI engineering stack also says that review judgments cite specific Codex rule IDs.
A rule record is an address for evidence, not a verdict.
Rather than saying “the API documentation is insufficient,” show which section of which document applies to which change, so the owner can verify the same evidence again. Cloudflare's published JSON is a shortened example, however, and should not be treated as a complete external configuration schema.
Try this now: create one non-blocking rule record
No public installation or authentication procedure for outside organizations has been identified for Cloudflare Engineering Codex or its internal review CLI. So the first exercise is not installing Cloudflare's tools. It is turning one recurring review comment into a traceable record using only your current team documentation and editor.
- Check the prerequisites and access. Prepare the exact section link for a current engineering standard you can open in a browser, one recurring review comment from a recent PR or MR, the rule's domain and document state, and an internal document or repository to store JSON. You do not need a separate Cloudflare account, installation, or authentication. You only need permission to open the internal source and save the record.
- Confirm the requirement level in the source. Compare the review comment against the exact sentence in the applicable standard. Classify it using RFC 2119's MUST and SHOULD definitions: is it an absolute requirement that must be followed, or a recommendation with valid exceptions? If you cannot find the currently effective source text or exact section, exclude it from automated-judgment candidates.
- Save the following input in your own document. This is an editorial review example based on Cloudflare's public fields; it is not a configuration file that runs at Cloudflare.
{ "rfc": "TEAM-API-001", "title": "API schema documentation", "status": "approved", "domain": "api-platform", "statements": [ { "slug": "api-schemas-must-be-documented", "section": ["API", "Schema"], "level": "MUST", "text": "External request and response schemas must be documented in the team's API specification.", "href": "https://internal.example/standards/api#schema" } ] } - Compare the source and record directly. Confirm that
hrefopens the exact section,textneither broadens nor narrows the source's scope, andlevelmatches the source. Setslugas a fixed identifier that does not overlap with other rules, so later judgments and exceptions can be linked. - Check the success state. Your first result is complete when the state, domain, stable slug, source section, requirement level, requirement statement, and direct link are all present and match the source in meaning. Keep
statusasapproved. At this stage, it does not block PRs or MRs.
Connecting real blocking requires additional pieces: a rule-reading evaluator, repository permissions, a result format, promotion authority, and a recovery path. Do not present an observation period or false-positive rate absent from public materials as though it were Cloudflare's official promotion criteria.
Before blocking, separate the check's role from platform authority
There is no need to hand rules that always produce the same answer from the same input to AI. Cloudflare provides custom linters for mechanically verifiable language-specific requirements and uses oxlint for TypeScript. The AI reviewer handles violations that require contextual reading, and the same review flow can run locally through an internal CLI. However, the external installation command and authentication method for that CLI have not been disclosed.
If you connect your own evaluator to GitHub required status checks, separate platform rules apply. According to GitHub's official documentation, a required check must be successful, skipped, or neutral before a merge can proceed, and you can restrict the expected source that updates the status to a specific GitHub App. This is not a procedure for reproducing Cloudflare's GitLab-based internal implementation; it is a separate implementation example available on GitHub.
Before promoting a rule to blocking, define at minimum who approves promotion, where judgments can be appealed, and who can clear an incorrect block. What makes AI review trustworthy is not more forceful wording, but a state you can explain: which rules are recommendations now, and which rules alone have blocking authority.
If you want to go deeper
How Cloudflare enforces engineering standards using AI — The primary case study for Codex domain ownership, the separation of RFC approval and enforcement, structured indexing, and code and spec review operations. blog.cloudflare.com
RFC 2119: Key words for use in RFCs to Indicate Requirement Levels — Explains the precise meanings of MUST and SHOULD and when to use strong requirement language. rfc-editor.org
About protected branches — GitHub's official documentation for configuring required status checks and their expected sources. docs.github.com
The Standard of Code Review — Public principles for distinguishing required changes from minor feedback and setting boundaries between continuous improvement and perfectionism. google.github.io
