This year I went from building web applications that occasionally called an LLM to designing and shipping a production AI assistant end to end. At Boldin I built the backend for Boldin Ai, an assistant that helps people understand and change their retirement plan, used by real people making decisions about real money.

What I really built is a harness. If you haven’t come across the term, an agent harness is all the software that sits around a model and turns it into an agent: the loop that runs it, the tools it can call, the context and memory it sees, and the guardrails that keep it safe. The usual shorthand is agent = model + harness. The model does the reasoning; the harness does everything else.

Retirement planning is an unforgiving domain for AI, and building a harness for it forced me to get hands-on with most of the techniques that separate a demo from a product. Here’s what I’ve worked with, and what I learned about each.

Agents and tool calling

I built a tool-calling agent runtime from scratch in Rails: the loop that lets a model choose a tool, validates its arguments, runs it, and feeds the result back until it can answer. The biggest lesson was to put existing, trusted systems behind the tools and let the model orchestrate rather than calculate. Models are excellent at deciding what to ask; they’re poor at arithmetic.

Multi-agent architecture

A single prompt with a growing list of tools degrades as it grows. I moved to a router with specialist agents, each with its own prompt, tool subset and model. It’s the same idea as splitting a monolith: smaller units that are easier to reason about, test and tune, and the freedom to put an expensive model only where the reasoning needs it.

Structured output and human-in-the-loop writes

Letting an AI change user data safely was the most interesting problem of the year. The pattern I landed on: the model never writes directly. It produces a proposal against a JSON schema, the server validates it, shows the user a diff, and the user approves it, with undo. The schema is both the contract the model writes to and the validation the server enforces. It’s a pattern I’d now use for any AI feature that takes action on someone’s behalf.

Retrieval-augmented generation

I built RAG pipelines over help content, video transcripts, and users’ own uploaded documents, using embeddings in Postgres with pgvector and HNSW indexes. The lesson most tutorials skip is that the index is a living thing. Keeping it in sync, and removing content that’s been retracted, matters as much as retrieval quality.

Context and memory

Long conversations need managing. I implemented a sliding context window with summarisation of older turns, plus explicit cross-conversation memory where the user can tell the assistant what to remember and what to forget. Giving users control over memory turned out to matter for trust as much as for quality.

LLM gateways, streaming and resilience

I built a gateway layer between the application and model providers that handles streaming, retries, per-agent model choice and automatic fallback when a provider fails. Streaming taught me something about infrastructure too. Long server-sent event responses don’t sit well on a traditional threaded app server, so I moved them onto a fiber-based server where a slow model costs an idle fiber rather than a blocked thread.

Sandboxed code execution

For calculations no existing tool covers, I gave the model a code-execution tool: it writes JavaScript that runs in an isolated V8 sandbox with strict time and memory limits and no network or filesystem access. Letting a model write code that runs in your production environment is mostly an exercise in what you take away from it.

Evals

New models arrive constantly, and without measurement every swap is a gamble. I built a custom eval suite from real usage patterns, so a new model can be tested against the questions that matter before it reaches a user. It turned model choice from opinion into a routine engineering change.

Guardrails and LLM-as-judge

I built a post-response validation layer: an LLM judge that checks each reply against the data the agent actually used, alongside fast rule-based checks for things like unsupported figures and broken links. Failed replies are regenerated before the user sees them.

Human escalation

No assistant gets everything right, so I built a path out: detecting when a user is stuck or the assistant is out of its depth, and handing the conversation to a human. Users don’t mind talking to an AI. They mind being stuck with one.

Feedback loops and observability

I ran a weekly triage of negative feedback and failed conversations, tracing each back to a root cause. Most of the year’s biggest quality gains came from that habit, and from small fixes no upfront design would have found.

MCP

Finally, I built an MCP server with OAuth 2.0, so people can connect assistants like Claude and ChatGPT to their own data with per-user, scoped access. MCP is quickly becoming the standard way to make a product usable from whatever AI someone already uses, and it rewards the same discipline as everything above: every write still goes through the same validation and approval path, whatever model is on the other end.

What ties it together

If I had to sum up the year in one line: the model is the easy part.

Calling an LLM takes an afternoon. The engineering is the harness around it: tools, schemas, approval flows, retrieval, gateways, sandboxes, evals, guardrails, escalation, and listening to users. That’s the work I’ve been doing all year, and it’s the work I enjoy most.