The most expensive default in AI engineering

If you’re building AI agents, you’ve faced this decision: which model actually powers the thing? GPT? Gemini? Claude? Mistral?

Here’s what usually happens. Nobody has time to test properly, so the team picks the most powerful model available. It works, everyone moves on — and the bill quietly grows. The frontier model becomes the default not because the use case needs it, but because nobody could prove it didn’t.

That proof is exactly what’s been missing. Vendor benchmarks tell you how models perform on someone else’s tasks. What you need to know is how they perform on yours — your requirements, your edge cases.

What we built

CodeValid turns your business requirements into a test suite for your agent, automatically:

1. Connect your Git repo — the one with your agent code. If your requirements live in a project management tool, connect that too for full business context.

2. CodeValid takes it from there — it sets up the test infrastructure, resolves dependencies, and generates test cases directly from your requirements. No test code written by hand. Every requirement becomes a verifiable check.

3. Pick your models and run — select any mix, small to frontier, and run the identical suite against all of them. Adding a new model is one click.

What happened when we tested a real agent

Take a look at our case study : Langchain text to sql agent evaluation

CodeValid AI agent testing is live now.

If you're building agents and want to know which model your use case actually needs? I'd love to show you, DM me or reach out at karthikh@codevalid.io

Here’s the thing: we still believe in software developers. We still believe in teams.

Yes, teams will get smaller. Radically smaller. But you cannot get rid of them, not for complex software. You still need someone who knows the process, domain expert. And every process can be implemented in multiple right ways, so you need someone who knows software and design, design expert, who can choose between those right ways, design, optimize, maintain, and you need someone who knows sales, marketing and customer service, business expert. And when your software has real users depending on it, one person isn’t enough. You need more than one to design it, to review changes it, sell it and to maintain it.

Now look at the tools everyone’s excited about. Cursor, Claude, Codex, they’re awesome, I use it everyday! but they’re built around a single developer. One person, one perspective.

We’re not building another development tool. That race is crowded, and honestly, it’s not the problem that keeps me up at night.

The problem is this: teams come together, developers, sales, customer service, all of them. And with AI, everyone on that team now has the flexibility to dictate a change to the code and get it done in an hour. That’s incredible. But now the question is: Can we get the same velocity of single person team? If so, how do you verify the software still does what’s right for the business, and keep the team aligned, when every person carries their own version of the business in their head?

So this is our bet. A world of super-optimized teams building complex software at incredible speed, and CodeValid as the alignment and verification engine underneath it all.

Making sure that what gets built is what the business required. Building Trust.

The arrival of cloud computing changed infrastructure.

Containers changed deployment.

Git changed collaboration.

Today, artificial intelligence is changing how software gets written.

What’s interesting isn’t that AI can generate code.

It’s how quickly that has happened.

A developer can now generate an API, build a React application, write database migrations, create unit tests, and deploy a working prototype before lunch.

That’s remarkable.

But it also exposes something the industry hasn’t talked about enough.

Software validation hasn’t changed nearly as quickly.

Most QA workflows still look surprisingly similar to the ones we built ten years ago.

Developers write code.

QA writes automation.

CI runs tests.

Failures are investigated.

Tests are updated.

Repeat.

That workflow still works.

But it’s beginning to show strain as software changes faster than humans can comfortably maintain automation.

Traditional Automation Solved Yesterday’s Biggest Problem

It’s worth remembering why Selenium became so successful.

Before browser automation, QA teams repeated the same manual scenarios every release.

Automation changed everything.

Later, Playwright and Cypress dramatically improved developer experience and browser reliability.

Those tools remain outstanding.

The problem isn’t Selenium.

The problem isn’t Playwright.

The problem is that the assumptions behind those tools were created in a world where humans wrote nearly all of the software.

Today, AI often writes the first draft.

The Bottleneck Moved

When software took weeks to build, spending days creating automation made perfect sense.

When software changes every few hours, that balance changes.

Engineering leaders are beginning to notice something interesting.

Developers aren’t waiting for code anymore.

They’re waiting for confidence.

Not because testing stopped working.

Because validation hasn’t accelerated at the same pace as development.

Five Areas Where Traditional Automation Feels the Pressure

Traditional automation still excels at browser testing, API validation, and regression testing.

But AI-generated software exposes challenges that didn’t exist before.

Automation maintenance. Every UI change creates script updates.

Specification drift. Software evolves faster than tests.

API explosion. Modern systems expose hundreds of endpoints.

AI agents. Agent behavior isn’t always deterministic.

Root cause analysis. Knowing a test failed isn’t the same as understanding why.

None of these are failures of Selenium or Playwright.

They’re symptoms of a changing software landscape.

A Different Way to Think About Validation

Most automation starts with writing tests.

A newer generation of software validation platforms starts somewhere else.

It starts by understanding the application.

Requirements.

Specifications.

Source code.

GitHub repositories.

Project documentation.

Application behavior.

Instead of asking engineers to manually describe every scenario, AI helps generate validation from the engineering context itself.

That doesn’t eliminate automation.

It changes how automation is created and maintained.

Why This Matters

I don’t think traditional QA automation is going away.

In fact, I expect Selenium, Playwright, Cypress, BrowserStack, and Postman to remain foundational tools for years.

But I also think we’re entering another transition.

The future probably isn’t:

AI replacing automation.

It’s:

AI helping engineers maintain confidence as software evolves.

That’s a very different problem.

And I believe it’s where software validation is heading.

This shift is one of the reasons we built CodeValid. Rather than asking engineers to spend more time maintaining automation, we believe software validation should begin with understanding the application’s context, its requirements, specifications, source code, APIs, and intended behavior. AI can then help generate, execute, and maintain validation so engineers can focus on delivering reliable software instead of constantly updating test suites.

Final Thoughts

For the past twenty years, software engineering has focused on writing code faster.

AI is rapidly solving that problem.

The next challenge isn’t generating software.

It’s knowing when that software is ready for production.

That’s why I believe the next decade won’t be defined by better test automation alone.

It will be defined by better software validation.

About the Author

Rakesh Shah is the co-founder of CodeValid, an AI-native software validation platform that helps engineering teams validate APIs, web applications, and AI agents using AI-assisted, context-aware validation. He writes about software quality, AI-assisted development, and the future of software engineering.

The Gap Between Knowing and Specifying

Domain experts possess a deep, intuitive grasp of their field’s problems often far deeper than any engineer. However, there is a massive difference between understanding a problem and specifying a technical solution.

Software doesn't just need to know what a good outcome looks like, it needs to know how data flows, how systems interact, and exactly what happens when things go wrong. Translating these understanding into a complete technical specification is hard. It’s not just about screens and buttons, it’s about how systems interact, how data flows, how edge cases are handled, and how decisions are encoded. Even with AI, incomplete or ambiguous specifications lead to software that “almost works,” but misses critical details that only surface later.

The Weight of Compounding Complexity

In the beginning, building an app feels like magic. But as soon as you add the second, third, or tenth feature, the “complexity tax” kicks in.

Every new button or data point introduces dependencies and unintended side effects. In software, complexity compounds. While an AI can generate a hundreds of lines of code in seconds, without good system knowledge it is hard to explain why a change in the User Profile broke the Payment Gateway. Understanding these ripples requires system-level thinking. Without it, a domain expert can quickly find themselves trapped in a “bug loop,” unable to reason through why their creation is suddenly falling apart.

The Hidden Half of Software: Infrastructure

Writing the code is actually only half the battle; the other half is keeping it alive.

To turn a script into a solution, you have to deploy it reliably, monitor it in production, manage failures, handle scaling, secure and compliant data, and control costs. While cloud providers promise simplicity, they often just trade one type of complexity for another. For a non-technical founder, the learning curve of infrastructure is a major barrier to entry. It’s hard to ship with confidence when you don’t fully understand the ground your software is standing on.

The Road Ahead

There’s no denying that software development is in the middle of massive change. In the last few years alone, we’ve seen more new tools, platforms, and approaches emerge than in the previous decade. Many of these are actively working to solve the exact problems above and make software more accessible than ever before.

If you’re a non-technical user who has tried to build software to solve a real problem, what challenges did you face along the way? Share your experiences and thoughts in the comments.

The Golden Age of Fast Development

A few years ago, shipping a new feature meant:

• Reading requirements carefully

• Designing APIs and data models

• Writing code line by line

• Manually testing multiple flows

Today?

Prompt → Code → Commit

With AI tools, developers can:

• Generate APIs in minutes

• Scaffold services instantly

• Write complex logic without remembering syntax

• Explore multiple approaches at lightning speed

Development has never been faster.

But there’s a catch.

Faster Code ≠ Safer Code

In real-world applications especially complex ones, features don’t live in isolation.

Every new feature interacts with:

• Existing business rules

• Data contracts

• User workflows

• Performance assumptions

• Security constraints

The Hidden Risk

When you add or modify one feature, you might silently break three others.

This risk existed before AI, but AI amplifies it.

Why?

• AI generates locally-correct code

• Systems require globally-correct behavior

A Very Familiar Story

Let’s say you ask AI:

“Add a discount feature for premium users.”

AI delivers:

• New DB fields ✅

• Discount calculation logic ✅

• API changes ✅

Looks perfect.

But later you discover:

• ❌ Existing invoices are miscalculated

• ❌ Refund flow is broken

• ❌ Reports don’t match totals

• ❌ Edge cases weren’t considered

Nothing crashed. Everything compiled.

But the system is now wrong.

A Real-World Example: Payments Feature Gone Wrong

Imagine a payments system in a production application.

Existing Flow (Working Fine)

• User places an order

• Payment is captured

• Invoice is generated

• Refunds and reports rely on the same data

Everything is stable.

New Feature Request

“Add partial payment support for enterprise users.”

You ask AI to implement it.

AI delivers:

• New payment_status values ✅

• Updated payment calculation logic ✅

• API changes for partial capture ✅

The feature works perfectly in isolation.

What Actually Breaks

A few days later, problems surface:

• ❌ Refunds return incorrect amounts

• ❌ Finance reports don’t match bank settlements

• ❌ Order status gets stuck in PROCESSING

• ❌ Edge cases fail during retries

No syntax errors. No crashes.

Just incorrect system behavior.

Why?

Because the AI optimized for the new feature, not for system-wide behavior.

Why Debugging Got Harder with AI

1. You Didn’t Write the Code

When AI writes code:

• You didn’t think through every branch

• You didn’t design every assumption

• You didn’t anticipate every edge case

Debugging becomes archaeology instead of engineering.

2. Confidence Without Understanding

AI-generated code often:

• Looks clean

• Follows best practices

• Uses correct patterns

This creates false confidence.

“It looks right, so it must be right.”

That’s dangerous.

3. Systems Are About Behavior, Not Code

AI is great at:

• Functions

• Classes

• APIs

But weak at:

• End-to-end flows

• Business intent

• Cross-feature impact

Your system breaks not because the code is bad, but because the behavior changed.

The Real Question Developers Must Ask

How do I know this AI-generated code is actually correct?

Not just:

• Does it work?

• Does it compile?

But:

• Does it satisfy the feature fully?

• Does it break existing flows?

• Does it introduce hidden bugs?

• Does it change system behavior?

This is where most teams struggle today.

What If We Had a Better Way?

Imagine a world where:

• AI doesn’t just write code

• AI understands system behavior

• AI checks existing flows before changes

• AI validates feature intent, not just syntax

Instead of asking:

“Write code for this feature”

We ask:

“Does this change break anything?”

That shift changes everything.

The Missing Layer: Confidence Engineering

What developers really need is not more code, but confidence.

Confidence that:

• New features don’t break old ones

• Modifications don’t alter intent

• Edge cases are covered

• Behavior remains consistent

This could come from:

• Automated behavior validation

• AI-assisted impact analysis

• Flow-based testing instead of unit-only tests

• Requirement-to-code traceability

Why Solving This Makes Developer Life Simple

If we solve this problem:

• Development stays fast

• Debugging becomes predictable

• Refactoring becomes safe

• Developers sleep better 😌

AI then becomes a true partner, not a risk amplifier.

Final Thought

AI didn’t make development harder.

It made our blind spots visible.

The next evolution isn’t faster code generation.

It’s knowing before production, what will break and why

Let’s Talk 👇

If you’re building real-world, complex systems with AI-generated code, this isn’t a future problem.

You’re probably feeling it already.

• A feature that worked perfectly… until it hit production

• AI-assisted change that broke an unrelated flow

• A merge that looked safe but caused silent data issues

💬 I’d love to hear from you:

• How do you deal with bugs introduced by AI-generated code?

• What’s the hardest part for you when reviewing or merging AI-written features?

• How do you gain confidence that nothing breaks?

Share your experiences or challenges in the comments 👇

Let’s learn from each other, and figure out how to make AI a true engineering partner, not just a fast code generator.

.