Building AI Agents: Lessons from the Trenches
Building AI agents that work in demos is easy. Building agents that work reliably in production? That’s where things get interesting.
Since 2025, I’ve built agents for everything from production workflows at Robynn AI to helping local businesses automate operational tasks. Here’s what I’ve learned. For a deeper cut on multi-agent architecture, see Why Four AI Agents Beat One Smart One and the Ollama priority queue.
What a production agent is
A production AI agent is a loop that calls tools, keeps state, and finishes real work under failure—not a chat demo that looks smart once.
The gap between those two things is almost entirely failure handling. A demo agent has one path: the one you showed. A production agent spends most of its code on the paths you’d rather not think about—the tool that times out, the API that returns a 200 with garbage in the body, the user who asks for something outside the mandate. Narrow job, explicit success criteria, and a human who can pull the handbrake.
The rest of this post is the specifics: how to pick the simplest architecture that fits, how to constrain outputs so you can trust them, and where to spend complexity versus where to just use something boring and proven. My defaults, if you want them up front: Python, structured outputs, LangGraph only when the control flow actually branches, boring datastores underneath.
The Agent Architecture Spectrum
There’s a spectrum of agent architectures, and choosing the right one depends entirely on your use case:
Simple Tool-Calling Agents
For straightforward tasks with clear inputs and outputs, you don’t need complex orchestration. A simple loop works:
def simple_agent(task: str, tools: list[Tool]) -> str:
messages = [{"role": "user", "content": task}]
while True:
response = llm.generate(messages, tools=tools)
if response.is_complete:
return response.content
# Execute tool calls
for tool_call in response.tool_calls:
result = execute_tool(tool_call)
messages.append({"role": "tool", "content": result})
This pattern handles 80% of use cases. Don’t over-engineer it. (This is part of my philosophy on boring technology—save complexity for where it matters.)
State Machine Agents
When you need deterministic control flow with AI-powered decisions at each step, state machines shine:
from langgraph.graph import StateGraph
workflow = StateGraph(AgentState)
# Define nodes
workflow.add_node("analyze", analyze_task)
workflow.add_node("plan", create_plan)
workflow.add_node("execute", execute_plan)
workflow.add_node("validate", validate_results)
# Define edges
workflow.add_edge("analyze", "plan")
workflow.add_conditional_edges(
"plan",
should_execute,
{"execute": "execute", "revise": "plan"}
)
Multi-Agent Systems
For complex domains, multiple specialized agents often outperform one generalist:
- Coordinator: Routes tasks to specialists
- Researcher: Gathers and synthesizes information
- Executor: Takes actions
- Validator: Checks results
The Patterns That Actually Work
1. Structured Outputs Are Non-Negotiable
Free-form text output from LLMs is unreliable. Always constrain the output:
from pydantic import BaseModel
class ActionPlan(BaseModel):
reasoning: str
actions: list[Action]
confidence: float
response = llm.generate(
prompt,
response_format=ActionPlan
)
In my experience, this alone eliminates half the production bugs I’d otherwise be chasing.
2. Retry with Exponential Backoff
LLM APIs fail. Rate limits hit. Network hiccups happen. Build resilience in:
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=4, max=10),
retry=retry_if_exception_type(RateLimitError)
)
def call_llm(prompt: str) -> str:
return llm.generate(prompt)
3. Observability from Day One
You can’t debug what you can’t see. Log everything:
- Input prompts (anonymized)
- Tool calls and results
- Token usage
- Latency breakdowns
- Error rates by category
I use a simple pattern:
@trace("agent_step")
def process_step(state: AgentState) -> AgentState:
with span("llm_call"):
response = llm.generate(state.messages)
log_metrics({
"tokens_used": response.usage.total_tokens,
"latency_ms": response.latency,
"tool_calls": len(response.tool_calls)
})
return state.update(response)
4. Human-in-the-Loop Escape Hatches
No agent should run completely autonomously in production. Always provide:
- Confidence thresholds that trigger human review
- Easy ways to pause and inspect state
- Clear audit trails
if action.confidence < 0.8:
await notify_human(
action=action,
context=state,
timeout=timedelta(hours=1)
)
Common Pitfalls
The “Just Add More Context” Trap
When an agent fails, the temptation is to add more instructions to the prompt. This usually makes things worse. Instead:
- Analyze why it failed
- Constrain the output format
- Add specific examples (few-shot)
- Consider splitting into sub-tasks
Ignoring Edge Cases
Agents in production see inputs you never imagined. That “simple” task of parsing customer emails? Wait until you see:
- Emails in multiple languages
- Forwarded threads with nested quotes
- Attachments described but not attached
- Sarcasm and ambiguity
Build defensive parsing and graceful degradation.
Over-Relying on the LLM
Not everything needs AI. A well-crafted regex or a simple rule often beats an LLM call for:
- Data validation
- Format checking
- Simple transformations
- Deterministic routing
Save the LLM for actual reasoning tasks. Use boring, proven tools for everything else.
What I’m Excited About
Four things I’m watching, in roughly the order I expect them to matter.
Standardized tool definitions—MCP and its neighbors—are the closest thing the space has to a portability story, and portable tools are what let you swap the model underneath without rewriting the agent. Right behind that: Haiku-class small models, which quietly change the economics of an agent loop. When a step costs a fraction of a cent, you can afford to run it three times and vote.
Memory and retrieval are becoming table stakes rather than differentiators, which is usually the sign that a capability is about to get commoditized into the SDKs. And evaluation is the embarrassing gap: we still have no good answer for testing agent behavior at scale, and everyone building this stuff knows it.
Common questions
Do I need a framework like LangGraph to build a production agent?
No. A plain tool-calling loop handles roughly 80% of use cases, and that is where I start every time. Reach for a graph framework when the control flow genuinely branches: conditional paths, different retry strategies per node, or state that has to survive across steps. Adopting it earlier just buys you an abstraction to debug on top of the agent you already have to debug.
Why do structured outputs matter so much?
Because free-form text is an unvalidated interface. Constraining the response to a schema, such as a Pydantic model, turns “the model said something weird” into a parse error you can catch, count, and retry. In my experience this one change eliminates about half the production bugs I would otherwise be chasing.
How many agents should a system have?
As few as the work requires. Split when the pieces have genuinely different failure modes and review standards, which is the argument I make in detail about Midas’s four-agent split. Keep one agent when the task is a single linear tool loop. Agent count is not a quality dial.
Should an agent ever run fully autonomously in production?
Not in my systems. Every agent I have shipped has confidence thresholds that escalate to a human, a way to pause and inspect state mid-run, and an audit trail. The escape hatch is not a lack of confidence in the model; it is the recognition that the cost of a wrong autonomous action is usually asymmetric.
Wrapping Up
Building production AI agents is part software engineering, part prompt engineering, and part systems design. It’s one of those domains where being a generalist actually helps—you need to understand the full stack, which is most of what the job looks like as a founding engineer. The agents that work best are:
- Simple where possible
- Observable always
- Constrained in their outputs
- Resilient to failures
- Honest about their limitations
If I had to compress all of it into one instruction for someone starting today: build the smallest agent that could possibly do the job, instrument it before you trust it, and add architecture only when a specific failure forces you to. Every complicated agent I’ve shipped got that way one justified step at a time. The ones that started complicated never worked.
