"Would my AI agent code actually pass the EU AI Act?" Someone got curious enough to scan 5,754 real files and find out.
The result: 97% failed. But the 23 files that passed had one thing in common that nobody planned for.
I thought this was legal's job
Whenever the EU AI Act comes up, everyone's first instinct is lawyers and paperwork. A recent Hacker News post flipped that assumption on its head.
A developer built an open-source static analysis tool called AIR Blackbox and ran it against 5,754 files across 11 well-known open-source AI projects with a combined 341,000+ GitHub stars, checking them against six EU AI Act articles (9, 10, 11, 12, 14, 15). Here's what came back.
If the story stopped there, it'd just be another "compliance is hard" post. But the author's follow-up comment was the real point. One of the files that passed all six articles was LiteLLM's auth module — and it wasn't written with compliance in mind at all. It just already had access control, structured logging, timestamps, and error handling.
As the author put it: "Teams are shipping AI agents into production with zero compliance infrastructure — not because they don't care, but because there's no tooling that makes it easy." In other words, what the EU is asking for isn't special legal knowledge — it's engineering hygiene most teams already know.
Which of the 6 articles actually trips people up?
Break it down by article and the gap is stark. Article 11 (Technical Documentation) passed 98% of the time, for a refreshingly boring reason — Python developers already write docstrings and type hints. Article 9 (Risk Management) went the other way: 97% failed. It checks for error handling and fallback logic around LLM calls, and most projects were built to just "work," not to fail gracefully.
| Article | What it checks | Result across 5,754 files |
|---|---|---|
| Article 9 | Risk management (error handling, fallbacks) | 97% failed — the worst of the six |
| Article 10 | Data governance (PII detection, input validation) | Not broken out individually |
| Article 11 | Technical docs (docstrings, type hints) | ★ 98% passed — the best of the six |
| Article 12 | Record-keeping (logging, audit trails) | Not broken out individually |
| Article 14 | Human oversight (approval gates, kill switches) | Not broken out individually |
| Article 15 | Accuracy & security (prompt injection defense) | Not broken out individually |
The author only published the best and worst of the six articles. The other four weren't broken out individually — but with an average of 2.2/6, most of them landed somewhere in the middle or below.
Someone else ran the same scanner against three whole frameworks. CrewAI and LangFlow passed 4 out of 6; Quivr, a RAG pipeline tool, managed just 1.
| CrewAI | LangFlow | At a glance | |
|---|---|---|---|
| Human oversight | 560-line human_feedback decorator | Guardrails + prompt injection detection | Both strong |
| Record-keeping | PASS | PASS | Quivr alone was WARN |
| Total score | 4/6 | 4/6 | Quivr scored 1/6 |
The takeaway: a compliance score is basically a proxy for engineering maturity. Projects that already had solid human-review hooks and tracing landed near the top without even trying.
Heads up
Some people used this scanner's results to file "please add this" issues on other open-source projects — and it didn't always land well. A request filed against the browser automation framework browser-use was closed as "not planned." Worth thinking twice before opening an issue on someone else's repo based on an automated scan.
This isn't just an EU story
The EU AI Act's high-risk system obligations become fully enforceable on August 2, 2026. Penalties run up to €35 million or 7% of global turnover, whichever is higher. If you serve EU users or put an AI system on the EU market, it can apply to you even if your company is based in Korea.
And Korea has its own version. The AI Basic Act took effect on January 22, 2026, making Korea the second country after the EU with a comprehensive AI regulatory framework. If your system counts as "high-impact AI," you get EU-style obligations around risk identification, assessment, and mitigation, with fines up to ₩30 million for violations. That said, Korea's science ministry has delayed enforcement by at least a year to give companies room to prepare, so this is a runway, not an immediate crackdown.
In short: the EU has a hard deadline of August 2. Korea is in a grace period, but the direction is already set. Fixing your code now covers both.
3-minute self-check — do it right now
- Install it
pip install air-blackbox. One line, no signup, no API key. - Run the scan
air-blackbox comply --scan. -vfrom your project root. Everything runs locally; your code never leaves your machine. - Check the weak articles first
Most projects fail Article 9 (risk management) and Article 10 (data governance) first. Start with error handling and input validation around your LLM calls. - Credit what you already have
If you write docstrings and type hints, Article 11 is basically already covered. Sort what's missing from what's already done before you prioritize. - Add framework-specific layers (optional)
Using LangChain or CrewAI?pip install air-blackbox[langchain]adds a dedicated trust layer.
Key point
Even the creator draws a hard line here: "This is not a certified compliance test. It is a starting point to identify potential gaps." Passing the scan isn't a legal get-out-of-jail card.
Go deeper
AIR Blackbox official site Pricing, supported frameworks, and install instructions in one place. airblackbox.ai
GitHub repo All 51 checks and the benchmark docs are public. github.com
10-second scan walkthrough Install to first results, step by step. dev.to
Three-framework comparison CrewAI, LangFlow, and Quivr scanned and scored article by article. medium.com
Korea's AI Basic Act guide Effective date, the definition of high-impact AI, and three obligations businesses need to know, explained by a law firm. help-me.kr
Compliance tool comparison A rundown of open-source and enterprise alternatives to AIR Blackbox. airblackbox.ai
