We are no longer discussing whether we should use AI or not. We are discussing how to use it so it has the biggest impact on our work. AI stopped being something we test. It became part of our daily workflow. Here are a few lessons that surprised us after a year of adopting AI across engineering.
AI Is an Amplifier
AI ain’t a silver bullet. Many people believe AI will magically improve their organizations. It won’t. AI is an amplifier. It won’t do magic. AI amplifies what already exists.
If you have crap in your codebase, AI will make it worse. If your tests are missing and your architecture is weak, AI won’t magically fix either of them. I’ve learnt it the hard way. I thought generating more code would automatically lead to faster delivery. Not true. Without proper processes, quality gates etc. you will end up in a worse place. And you will get there quicker because of AI.
Bottlenecks Don’t Disappear. They Move.
The second thing that surprised us was that bottlenecks don’t disappear. They move. Previously “coding” was a bottleneck.
We thought AI had solved the problem. Funny. Once we optimised the coding part, the bottleneck moved and we started saying the the work is getting stuck in idea (“problem definition”) or post coding, in review and testing.
Even though AI changed many things, it didn’t change theory of constraints 🙂
We Still Haven’t Found the Right AI Metric
After almost two years of AI adoption, we still haven’t found the right metrics that could tell us how we use it. Traditional metrics suggest that delivery is getting faster but but they don’t tell us how much of that improvement comes from AI. There is no direct correlation with AI. Companies are spending millions on AI and when their CFO comes asking
What is the ROI on the AI?
we have no idea how to answer it. The problem becomes even bigger when we hear that 95% of of AI pilots fail. Is it because AI isn’t that good? I doubt it. Part of the problem is unrealistic expectations. Another part is the lack of meaningful measurements. The question I’ve been struggling with recently is:
Do we still need such measurements in the era of AI? Maybe we need to rethink the rules entirely?
I dunno.
Human-in-the-Loop Came Back Much Faster Than Expected
At the beginning, many people predicted that we are doomed, AI will take over and we will loose our jobs. The assumption was that people would no longer be needed because AI would know what to build and how to build it. Shortly after, we realised that to achieve the best results, we need to have human-in-the-loop to to review and validate AI-generated work. Despite everything AI can already do, taste, judgment and having an opinion is still on our side. Is this true in every context? Of course not. The context still matters. There are certainly areas where AI can operate on its own.
The Biggest Changes Happened Outside the Codebase
This brings us back to the Theory of Constraints and our SDLC. The fact that we are able to deliver more features in shorter period of time, means that AI revolution is also outside of codebase. Many companies are moving towards builders, Forward-deployed-engineers, or whatever-we-call-them to get engineers closer to clients. Their role is to talk to customers, understand their problems, make decisions and use AI agents to implement solutions. This fundamentally changes how requirements are created. The role of PM is also changing because of this revolution. Also, outside of codebase, many mature companies ask their engineers to get more ownership of the solutions they build. Beyond the technical changes, it’s also a big organisational change in many companies.
AI shortened the distance between customer problems and implementation.
Wrapping up
Looking back, almost none of the lessons were about AI itself. They were about us. About how we plan work, communicate, review code, measure success and make decisions.
AI didn’t eliminate bottlenecks. It moved them.
It didn’t replace engineers. It changed what good engineering looks like.
It didn’t magically improve our organization. It exposed both its strengths and its weaknesses much faster than before.
That’s why I believe we’re entering a new chapter of software engineering. The biggest competitive advantage won’t come from using the newest model or the most advanced agent. It will come from building organizations that know how to think, adapt and make good decisions.
AI may be writing more code than ever before, but the scarcest resource is no longer code. It’s judgment.
And that’s probably the most important lesson we’ve learned over the past year.