Why AI-Built Apps Pass Every Test and Still Break in Front of Users

Your coding agent builds the whole app and gets the suite passing in an afternoon. But those tests ran against a handful of placeholder rows, or nothing at all, so the bugs that only show up against real data walked straight through. Photo by Sunder Muthukumaran on Unsplash Last month I handed a coding agent a one-line brief and let it run. It wrote the migrations and wired the routes on top of them, and by the time it stopped, pnpm dev came up clean and the test suite was green on the first try, then I clicked around a build that rendered every page perfectly and had nothing on any of them. The dashboard reported zeros across the board, the orders table held no orders, and the "recent activity" feed was a blank rectangle where the demo was supposed to live. What got me is that the schema underneath was genuinely good — real foreign keys, a couple of unique constraints, a CHECK or two holding the shape togetherf. The agent had done the hard structural work and stopped one step short of what makes an app testable at all, which is getting real, connected data into those tables in the first place. For a long time that gap was a corner case, the kind of thing you’d hit once and shrug off. like it has quietly turned into the default. In June 2026 Supabase reported that more than 60% of new databases on its platform are now launched by some kind of AI tool, with agents deploying the majority of them and Claude Code the largest single contributor since the start of the year — these agents stand up the database and apply the migrations, and more and more they write the whole application on top of it. What they are worst at is the unglamorous step underneath all of it, the one where the tables fill with data that behaves the way real data does. “Just tell the agent to seed it” So you ask for it. Insert some sample rows, t he agent obliges, and that’s exactly where the trouble starts. It fills those tables one at a time, writing INSERT statements in whatever order the schema file happens to declare them. If orders sits above users in that file, the orders go in first, and Postgres throws out the very first row because its user_id points at a user that doesn't exist yet. At that point the agent improvises a fix — it might drop the offending constraint, or stuff user_id with a plausible-looking integer, or shuffle the INSERT order by trial and error until the errors stop scrolling up the terminal. Eventually the errors stop and every table has rows in it. None of those rows add up to a graph, though, and a single query is enough to prove it: SELECT count(*) FROM orders o LEFT JOIN users u ON u.id = o.user_id WHERE u.id IS NULL; -- orders pointing at users that were never generated Anything above zero means the agent handed you type-correct noise dressed up as data. Every user_id validates on its own, a real integer sitting in a believable range, while pointing at a user that was never generated. Empty tables make your tests lie There’s a quieter version of this failure, and it’s worse precisely because nothing turns red. Point a test suite at three hand-seeded rows, or at a table with none, and it goes green, though the green certifies only that the data was too thin to contradict the code. Pagination has no second page to break on, because actually there is no second page, and the N+1 query that would crawl in production returns in a millisecond here, walking its whole five rows. To be honest, I’ve shipped a bug exactly like that quite many times. The test suite was green and coverage looked healthy right up until the feature fell over at the first afternoon it met a table with a few hundred thousand rows after weeks of tests quietly agreeing with an empty database. Not all of what slips through this way is a performance problem. Authorization is the sharpest example — a query meant to scope results to the current user only gets tested when a second user’s rows are sitting in the same table, so seeding a single user leaves the endpoint that would hand back someone else’s record by ID looking completely correct, with no other record around for it to leak. The same emptiness hides tenant-isolation bugs and permission checks that never fire, because the data that would trip them was never there. What “fill it correctly” actually takes When the table-by-table INSERT breaks, it isn't breaking because someone was careless. The break is structural, and the structure is where the real work hides. Ordering the inserts is the thing that looks like it should be the first. Rows have to land parents-before-children, each one arriving only after the rows it references already exist, which is almost never the order the schema file happens to list its tables in. When two tables reference each other in a cycle, one side goes in first with its back-reference left blank, and a second pass fills that reference once both rows exist. Constraints are where it gets fiddly. Each one has to hold while the rows are being generated, not patched in afterward once the errors have scrolled past: for example, a UNIQUE(email) column needs a generator that remembers every address it has already produced, so the next one won't collide with an earlier one. Hand that same generator a CHECK (ship_date >= order_date) and the two dates suddenly have to be produced together, by something that understands they're related, because drawn as independent random values they'll violate the rule about half the time. Even with order and constraints handled, the rows still have to make sense together, and that coherence is the thing most tools skip and the thing that gets you caught: user’s signup timestamp ought to fall before their first order, row that reads status = 'shipped' while carrying no shipped_at is nonsense on its face, and a city that disagrees with its postal_code is exactly what a reviewer clocks two seconds into a live demo, long after a column-at-a-time generator has pronounced every field valid on its own. This is where Faker and everything built like it come apart, because each value gets drawn from its own distribution with no glance at the value sitting next to it, so every field can look impeccable while the row as a whole quietly makes no sense. None of this is hard in the sense of being clever. It’s fiddly and repetitive and easy to get subtly wrong, which makes it exactly the kind of work you don’t want a probabilistic agent improvising at 2 in the morning against a schema it only half-remembers. The landscape, once you stop hand-holding the agent Once you decide the data deserves a real step in your pipeline, the options fall into three rough tiers. When you need a column of believable names, or a throwaway script against a couple of tables, the free and DIY route is enough. That means Faker and its ports across languages, plus whatever SQL you hand-write around them; In short, it stays blind by design to the adjacent column and to the table across the foreign key, which leaves every relationship you care about for you to wire up by hand and keep working. Neosync (closed since August 2025) and Seedfast sit one tier up, the schema-aware generators. Both read the live schema and produce data that keeps every foreign key intact, so the insertion order is never something you spell out by hand; where they mostly differ is in how much of your domain you describe up front and how much they read straight from the schema. This is the tier built for exactly the empty-tables problem the agent left behind. For regulated organizations, the ones that have to mask production data and keep an audit trail alongside whatever they generate, there are the enterprise platforms like GenRocket and Delphix . Which one to reach for When an empty database is fine Not every empty table is a problem worth solving. Three commits into a prototype, with the schema changing by the hour, seeding is wasted motion; the agent’s five throwaway rows are plenty to confirm a page renders and a route responds, and you’re better off skipping the ceremony and moving on. Stability is what changes the math. Once the schema settles and someone wants a realistic demo, or the first test starts to lean on more than one row existing, that empty database becomes a real problem — the thing a reviewer or a customer runs into first. Picking them up on purpose, before a test or a demo quietly starts depending on it, is the part worth owning yourself. Why AI-Built Apps Pass Every Test and Still Break in Front of Users was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.