I Benchmarked My AI Coding Agent Against Human-Written Code. It Won Every Metric but One

A self-refining Gemini-powered agent, five real-world Python tasks, a +12.2 Maintainability Index gap — and the one metric where human code still came out ahead. There’s a debate happening in every engineering Slack channel right now: is AI-generated code actually good, or does it just look good? Most of the takes are vibes. Someone pastes a ChatGPT snippet, someone else finds a bug in it, and everyone retreats to their prior opinion. I wanted numbers instead. So I ran this as a proper study — formal research questions, defined quality thresholds, threats-to-validity section, and all. I built an autonomous agent that writes, runs, and refines its own Python code, then put its output through the same static analysis gauntlet we’d use on a human developer: Pylint, Radon’s Maintainability Index, cyclomatic complexity, and bug density. Then I ran human-written implementations of the same five tasks through the same pipeline and compared them. The agent won on almost everything. Almost. Why “the agent felt smart” wasn’t good enough I’ve built agents before — if you read my council of 18 models piece, you know I like making AI systems argue with something, even if it’s themselves. But “it produced working code” is a low bar. Working code that nobody can maintain is how legacy systems are born. So I framed three research questions before writing a line of the framework: RQ1: Can an AI agentic framework achieve code quality comparable to that of human developers in Python? RQ2: What are the relative strengths and weaknesses of AI-generated code across different types of tasks? RQ3: How much do the framework’s tools — web search, automated library installation — actually contribute to functional correctness? I also set pass/fail thresholds up front so that I couldn’t move the goalposts afterward: Maintainability Index above 20, Pylint above 8, bug density below 0.1 defects per line. The architecture: an agent that reads its own terminal The core loop is simple to describe and annoying to build: the agent doesn’t just generate code — it executes it, reads the terminal output, and decides whether the result meets the requirements. If not, it loops back with the reason for failure. Figure 1: The self-refining agent loop — design, generate, execute, evaluate, repeat. Five components make it work: Code Designer: turns the task into explicit technical requirements the script must comply with, before any code is written. This step matters more than the model choice. Code Generator: Gemini Flash produces the implementation. Why Gemini? Honest answer: because it’s free (more free quota than others). That’s not a bug in the study — it’s the point. If a free model inside a good feedback loop beats human code, the framework is doing the heavy lifting, not the LLM. Script Evaluator: a second agent that runs the script and analyzes the outcome. Fail: back to the generator with the failure reason. Pass: ship it. Web Search Tool: lets the generator fetch current documentation when it hits a wall. (Remember this one — it’s the hero of the story later.) Library Installer: resolves dependencies in an isolated environment, so a missing package never kills a run. The design principle: the model is replaceable; the feedback loop is not. The benchmark: five tasks, one pipeline, no cherry-picking Both the agent and the humans implemented the same five single-file Python programs: a basic CLI echo utility, a JSON-backed to-do list manager, a CSV filtering tool, a logging HTTP server, and an interactive AI chat client (REPL, system-message switching, syntax-highlighted output). Every script — AI and human — went through the same automated analysis: Pylint for convention and error-proneness, Radon for Maintainability Index and cyclomatic complexity, plus logical lines of code and bug density. The results The full scorecard: five metrics, averaged across the five tasks. Agent 4, humans 1. The agent won four of the five metrics — Pylint 7.74 vs 7.27, Maintainability Index 76.59 vs 64.39, bug density 0.30 vs 0.43 defects/KLOC, and 59.8 vs 95.6 logical lines of code. The humans took exactly one: cyclomatic complexity, 1.63 vs 3.91. Hold that thought. Three things jumped out at me. The Maintainability Index gap is not subtle. +12.2 points means the agent’s code was consistently more modular and more self-documenting. Reviewing the outputs manually, the reason was obvious: the agent decomposes by default . It writes small named functions with docstrings because that’s the statistically dominant pattern in its training data. A human under time pressure writes the nested loop that works and moves on. I know, because I am that human. The agent says the same thing in 37% fewer lines. 59.8 logical lines versus 95.6 for functionally equivalent programs. Less code is less surface area for bugs — which the bug density number backs up. And the gap explodes on the hardest task. The averages actually understate what happened on the AI chat client, the most complex of the five programs. The human version: 387 logical lines, a Maintainability Index of 28.66 — barely above the “legacy code” floor — and one function with a cyclomatic complexity of 23 . The agent’s version: 100 lines, MI of 68, max complexity of 12. Same functionality. The human REPL wasn’t wrong; it was the kind of code that works today and terrifies whoever inherits it in six months. As task difficulty went up, the human code got monolithic. The agent’s didn’t. The one metric where human code won The agent’s average cyclomatic complexity was 3.91 against the humans’ 1.63. More than double. At first glance, that looks like a loss for the AI. Cyclomatic complexity counts independent paths through the code — lower value usually means simpler. But when I read the scripts side by side, the story got more interesting. The human code achieved low complexity by being monolithic : long straight-line procedures with the occasional loop. One human script literally scored a complexity of zero — no branching at all — while carrying a bug density of 0.75, the worst in the entire study. The agent’s code had higher per-function branching precisely because it split logic into many small, single-purpose functions, each handling its own edge cases and error paths explicitly. So which is actually better? The Maintainability Index — which folds complexity, volume, and structure together — says the agent’s trade-off wins. But it’s a genuine trade-off, not a knockout, and it’s exactly the kind of thing a single metric would hide. If you only measured cyclomatic complexity, you’d conclude AI writes worse code. If you only measured Pylint, you’d conclude it’s barely better. The truth needs the full panel. (I’d genuinely like to hear where readers land on this — is explicit branching in small functions better or worse than long simple procedures? Fight it out in the comments.) The finding that surprised me most: the tools don’t show up in the metrics Here’s the RQ3 result I didn’t expect. I correlated the framework’s tool usage — web search and automated library installation — against the quality metrics. Neither tool improved a single static metric. Not Pylint, not maintainability, not complexity. If you only looked at the static analysis, you’d conclude web search and dependency installation are dead weight and delete them. You’d also be deleting the only reason the code runs . The static metrics measure how code is structured. The tools determine whether it’s functionally correct at all — whether the API call uses the current library interface, whether the dependency actually resolves. Structure and correctness turn out to be nearly independent axes, and most “AI code quality” debates conflate them. An LLM alone gives you well-structured code. The agentic scaffolding is what makes it work . The moment the agent fixed its own math The test that convinced me this approach has legs wasn’t any of the five benchmark tasks. It was a domain-specific stress test in signal processing. The exact prompt I gave the agent: “Create a Python script that implements a notch filter, use synthetic data as sum of sines 0.1Hz and 0.55Hz. Filter should remove/reduce the power of the 0.1Hz sine by at least 90 percent. You cannot access data files, do everything within the script. Plot and save necessary figures.” This is the kind of task where LLMs classically faceplant — it requires getting filter coefficients numerically right, not just syntactically plausible. And on the first pass, the agent’s filter didn’t attenuate properly. Here’s where the web search component earned its place in the architecture. The evaluator flagged that the output missed the attenuation requirement. Instead of blindly regenerating, the agent searched for documentation on the coefficient calculation, corrected its math, and re-ran. Final result: the 0.1Hz component is nearly erased from the spectrum — over 90% power reduction at the target frequency , exactly as specified. Figure 2: Magnitude spectrum and time series of the agent’s final filter. The 0.1Hz peak is nearly gone; the 0.55Hz signal passes untouched. No human intervened between “wrong filter” and “working filter.” That closed loop — execute, detect failure, retrieve knowledge, correct — is the difference between a code generator and a coding agent. The uncomfortable arithmetic Now, the part everyone actually argues about. A junior developer runs roughly $5,800 a month. This agent, including every API call and the infrastructure around it, runs under $1,000 — and once built, costs closer to 5% of that salary. It’s also faster, doesn’t lose focus, and its measured output quality is what you just saw. I want to be careful here, because the lazy conclusion — “juniors are obsolete” — is wrong, and my own data shows why. Every task in this study was specified by a human . The Code Designer works because someone decided what to build and what “correct” means. And the humans still hold clear ground that the metrics can’t capture: creativity, visual and UX judgment, and deep domain knowledge when it exists. What the numbers actually suggest is that the junior role is shifting upward: less “implement this ticket,” more “specify, review, and verify what the agent implemented.” That’s a different skill set — and honestly, a more interesting job. Honest limitations Before anyone cites this in a LinkedIn hot take, this benchmark covers small, single-file Python utilities with no disk I/O — the findings may not transfer to large multi-module systems or other languages. The human baseline comes from a limited group of programmers, not a representative industry sample. And static metrics measure structure, not architectural judgment — Pylint can’t tell you whether the abstraction is right , only whether it’s tidy. These results are a data point in the debate, not the end of it. That’s also exactly why the repo is public: same tasks, same analysis script, reproducible pipeline. If you think your code beats the agent — run it through and prove it. What’s next Three upgrades are already in motion, and each will get its own write-up: automated PyTest generation , so the agent is judged on dynamic correctness, not just static structure; richer metrics — coverage, mutation score — fed back into the refinement loop so the agent improves against them progressively; and the big one, scaling from single scripts to full projects via a divide-and-conquer module that decomposes a codebase into script-sized units, dispatches them to specialized sub-agents, and recombines the tested pieces. I’m also planning to swap Gemini for stronger models to isolate how much the LLM itself matters versus the loop around it. Try it yourself The full framework, both code sets (AIWritten/ and HumanWritten/), the quality reports, and the analysis automation are all in the repo: → github.com/Alpsource/SQM_Test bash git clone https://github.com/Alpsource/SQM_Test.git pip install pylint radon bandit google-genai export GOOGLE_API_KEY='your_key_here' If you found this useful, a ⭐ on the repo genuinely helps, and I write one deep-dive like this roughly every two weeks — from building I-JEPA from scratch to hybrid A*+RL flight agents . Follow along, and tell me in the comments: would you merge a PR if you knew no human wrote it? I Benchmarked My AI Coding Agent Against Human-Written Code. It Won Every Metric but One was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.