As engineering consultants here at Axian, clients often ask us to parachute into mature products with years of development (and technical debt). Navigating these codebases is difficult and time-consuming. It is critical that we work with existing staff to help guide us through the process. However, as codebases mature and staff are asked to deliver features, blind spots can form, making it more difficult to accurately characterize gaps against a well-architected framework.
Thanks to recent advances in AI and our ability to self-host with strict security controls, we can keep client code within our ecosystem. This unlocks all new opportunities, such as building a bot to guide discovery and help us find issues as quickly as possible, with no added risk. We created the WAR Bot (Well Architected Review Bot), a tool that helps us focus on the most pressing needs. Timing is critical, and the WAR Bot shortens the path to the right questions without affecting our rigorous reviews.
Why now?
We recognized the perfect opportunity to build the tool:
- The urgency has never been greater; our clients need us to identify more complex issues, with more complex codebases, in less time.
- Larger, better-trained AI models and improved tooling have raised the bar on what’s possible.
- Self-hosting with strict controls allows us to safely manage client code, addressing common technical and legal concerns. Clients trust us to hold their data as we innovate more powerful ways to discover and address issues.
These shifts meant we could test WAR Bot in a controlled way, with client code kept inside our walls.
The journey
Building the WAR Bot was an iterative process:
- Choosing a codebase: We selected a codebase we had already reviewed through our standard discovery process so that we could directly compare results.
- Starting simple: We began development very simply. We asked a simple chatbot to review the codebase. We quickly learned what didn’t work: generic guidance to an AI didn’t accelerate decisions and occasionally sent us down blind alleys.
- Iterating prompts: Drawing on our experience crafting LLM prompts and the discovery tooling we already trusted, we treated WAR Bot like a product. We strengthened product requirements, continually clarified the outputs we need, aligned the questions to our code review framework, and measured the usefulness of insights with each iteration. Each pass reduced noise and increased signal, asking more of the right questions, sooner. Most importantly, it provided clearer next steps for our consultants.
This work revealed how WAR Bot could materially change the discovery experience.
What WAR Bot means for discovery
WAR Bot transforms our discovery process:
- Speed: As a first step, WAR Bot helps us much more quickly discover issues, freeing time for in-depth analysis and guidance that is high-impact and specific.
- Focus and fit: It helps us ensure that findings are relevant to the client’s most critical needs. With WAR Bot assisting us in characterizing the codebase, we can focus our efforts where problems are most likely, and on the other pillars of a well-architected framework, delivering more precise findings.
- Less invasive: We reach answers and focus points with less time required from busy client staff.
WAR Bot provides a heatmap of the system. It helps us focus on the areas that might be interesting.
How we’ll use it
This naturally raised the question: how does this fit into the way we work today?
- Alongside our standard process: For initial rollouts, WAR Bot runs in parallel with our current discovery process. It helps us measure consistent improvements and understand where it unlocks capability.
- Human-verified by design: WAR Bot surfaces technical leads; Axian consultants investigate, confirm, and craft the remediation steps. We can never fully rely on an LLM to do our homework.
- Iteratively: We will keep refining prompts and usage patterns when WAR Bot underperforms—and direct staff to do the legwork when needed.
Guardrails and limits
To make these benefits reliable and repeatable, we operate with clear guardrails.
- Scope: WAR Bot is a tool to analyze code. Well-architected reviews also span business and development processes, which require people.
- Ownership: Humans are responsible for signing off on the review. WAR Bot assists; it does not replace expertise.
- Coverage discipline: We still examine all the pillars of a well-architected framework; WAR Bot helps prioritize, not decide.
Closing thought
We’re excited to take WAR Bot into future discovery. Early use will focus on “how does this help me?” and “how do I get the most benefit?”. It isn’t a sweeping change to the way we work. It’s a guided starting point that unlocks better analysis, faster, while keeping experts firmly in the loop.