When a scan result looks like a legal determination
If you may provide an AI agent in the EU, it is natural to ask, “Will our code pass the EU AI Act?” A tool that scans files and shows a PASS for each article can be appealing. It can seem like a quick way to find risky areas before a legal review.
The creators of AIR Blackbox reported scanning 5,754 Python files across 11 open-source projects, with an average score of 2.2/6, and said only 23 files received PASS for all six articles. For Article 9, 97% of files did not contain the signals they mapped; the figures were 89% for Article 12 and 84% for Article 14.
But you should not read this as “The EU AI Act inspected code and failed 97% of it.” It was neither an EU authority investigation nor a legal conformity assessment. It was the output of file-level heuristics defined by the tool’s creators.
What 23 PASS results actually mean
In that benchmark, PASS for an individual article meant that at least one of several sub-signals linked to that article was found in a file. The 23 files met this condition for each of the six articles; they were not found to be legally compliant with six EU AI Act articles.
At the time of the benchmark, the creators linked risk classification, access control, and risk auditing to Article 9, and error handling and testing to Article 15. The LiteLLM authentication module they used as an example already had access control, structured logs, timestamps, and error handling. The takeaway is not that “the 23 files had a special secret,” but rather that everyday operational controls can surface as useful clues in static analysis. Public materials do not let us verify what all the remaining PASS files had in common.
These figures are difficult to reproduce as a benchmark.
The repository-specific commits, file-selection criteria, and raw JSON used for the scan could not be reproduced through currently public paths. We also could not confirm an independent evaluation of false-positive and false-negative rates. So 2.2/6 or 0.4% should not be used as a passing threshold for your project or an industry average.
The law looks at systems, not files
The relevant EU AI Act obligations are broader than a single code pattern. Article 9 requires a risk-management system that is performed iteratively and documented throughout the entire lifecycle of a high-risk AI system. The technical documentation in Article 11 is not a docstring or type hint; it is documentation prepared before placing the system on the market and kept up to date so authorities can assess the system’s conformity.
| Article | System-level verification required by law | Key clues sought by the v1.15.0 scanner |
|---|---|---|
| Article 9 | Risk identification, assessment, mitigation, and periodic review | Error handling, fallback·recovery |
| Article 11 | Current technical documentation enabling conformity assessment | Documentation·type information |
| Article 12 | Ability to automatically record events throughout the system lifetime | Logging·audit-trail calls |
| Article 14 | Human oversight means to disregard·override output or safely stop the system | Human-in-the-loop, usage·budget controls |
| Article 15 | Accuracy, robustness, and cybersecurity | Retry·backoff, prompt-injection defenses, output validation |
The right-hand column summarizes the currently public v1.15.0 implementation and differs from the mapping used when the 5,754 files were scanned. The current code scanner places error handling and fallback·recovery under Article 9; human-in-the-loop and usage·budget controls under Article 14; and retry·backoff, prompt-injection defenses, and output validation under Article 15.
A logging call does not prove a retention period or auditability. An approval function does not show whether an operator can intervene in time. The scanner is best used as a navigator that points to places to verify, not as evidence. The repository also states that the tool is not a certified compliance test and is a starting point for finding potential gaps. Its self-assessment covers 72 prepared fixtures and 12 checks; it does not guarantee accuracy for arbitrary production code.
Start the first scan by separating the sources of results
You need Python 3.10 or later, local source for the project to inspect, and permission to install packages and run commands. The process below is not a legal compliance determination; it is work to find initial control clues in code. It is described against the verified v1.15.0 distribution, but check conditions and output may change with updates.
- Install the package in the project environment.
pip install air-blackbox - Record the installed version and help output first. Include the package version and execution date in the scan report, and check options in the installed command’s help. Do not apply the article mapping from the historical benchmark directly to current output.
- Move to the project root. For a production repository, start from a separate branch or a read-only copy.
- Run the first comparison with optional analysis and history saving disabled.
air-blackbox comply --scan. -v --no-llm --no-save
The v1.15.0 default command checks local gateway status and outputs both static and runtime checks. It can run AI analysis when Ollama and the default model are available, and it saves check history locally under the default configuration. These options exclude optional LLM analysis and history saving, making initial static results easier to compare. - Separate a disconnected gateway from an absent control. Static checks may continue without a localhost:8080 gateway. If runtime checks report no observation data, record that separately as runtime verification has not yet been performed, not as evidence that controls are absent.
- Save the status and evidence wording for every check together. Instead of the historical article-level 6/6 score, the current CLI outputs PASS·WARN·FAIL for each individual check. PASS does not always mean that a control signal was found. For example, the Article 9 error-handling check can pass conditionally when it finds no direct LLM call. Do not copy the status alone; read the check conditions and evidence too.
- Link only items with reported locations to files. Some checks show aggregates such as “found in N files” instead of a filename or line location. Do not record a result without a location as though the code location were automatically verified; perform separate code searches and manual review based on that check’s detection patterns.
- Compare the results with system evidence. PASS·WARN·FAIL are not legal determinations. Verify that controls work in the risk register, current technical documentation, actual log retention, human stop authority, and accuracy·robustness·security tests.
- Finally, classify the scope of application. Separately assess whether the system falls within a high-risk category, whether your organization is a provider or deployer, and which implementation timeline applies.
The first success criterion is not 6/6. The first stage is complete once you have saved the installed version and execution options, each check’s status and evidence, and linked only items with an actual reported location to the corresponding file. Separate location-less aggregate results and checks with no runtime data into a follow-up search and operational-verification list.
EU relevance and effective dates also need separate review
Even operators outside the EU may fall within scope if they place or provide an AI system on the EU market, or if its output is used in the EU. But you cannot reach a blanket conclusion just because there is one EU user. Consider provider·deployer roles and the actual market-provision relationship together.
The implementation timeline should not be reduced to a single date, August 2, 2026. While the Act generally applies from that date, current official guidance distinguishes specific Annex III high-risk areas as December 2, 2027, and Annex I high-risk systems embedded in regulated products as August 2, 2028. Prohibited practices, AI literacy, and general-purpose AI obligations have other timelines.
Fines also vary by violation type. Up to €35 million or 7% of worldwide annual turnover is the cap for violations of prohibited practices under Article 5. The cap for violations of general operator obligations, including Article 16, is €15 million or 3%; for SMEs, the lower of the monetary and percentage caps applies. That is why an Article 9 warning should not be immediately equated with a “7% fine risk.”
According to the official legislative history, Korea’s Framework Act on Artificial Intelligence took effect on January 22, 2026. But an EU scanner result cannot determine whether Korean law applies. Domestic classifications and obligations must be reviewed against separate official legislation.
If you want to dig deeper
Consolidated text of Regulation (EU) 2024/1689 Check the current wording of Articles 2·9·11·12·14·15·99 to see the difference between static-analysis signals and system obligations. eur-lex.europa.eu
Official AI Act policy guidance Useful for checking application dates by obligation and the high-risk system schedule. digital-strategy.ec.europa.eu
airblackbox/airblackbox The official repository for checking the current installation command and the limits of static gap analysis. github.com
AIR Blackbox v1.15.0 source Shows how the current CLI handles static·runtime checks, optional Ollama analysis, and local history storage. raw.githubusercontent.com
