It is a Tuesday evening change window at a mid-sized insurer. The policy-service, a Spring Boot application that has run quietly for years, just shipped its first AI feature: an assistant that helps call-center agents answer coverage questions. Behind the scenes, Spring AI's ChatClient sends the question to a model, the model calls a few tools to look up the policy and claims history, and one tool creates a callback task in the CRM when a customer needs a follow-up.
The platform team starts a routine rolling restart. Pods drain, new pods come up, health checks go green. By morning, two things show up in the support queue. Several call-center agents saw their assistant fail mid-answer, with no way to resume. And a handful of customers now have two identical callback tasks, because a retry ran the whole interaction again, including the tool that had already created the first task.
Nothing in the logs looks like an outage. Every component did what it was designed to do, under assumptions built for a different kind of workload.
Why LLM calls break Spring's reliability model
Spring teams in banks, insurers, and telcos have strong reliability habits. You wrap database work in @Transactional, annotate flaky integrations with @Retryable from Spring Retry, and rely on graceful shutdown and mature ops tooling. Those habits work because most calls in a Spring service are short, mostly deterministic, cheap to repeat, and either transactional or idempotent.
An agent interaction violates almost all of those properties at once.
- It is long. A single ChatClient call that triggers several tool calls can take tens of seconds or more. That often exceeds the pod shutdown grace period.
- It is non-deterministic. Run the same prompt twice and the model may pick different tools, in a different order, with different arguments.
- It is expensive. Every repeated model call costs tokens and latency.
- It is external. The model provider and the tools it calls live outside your transaction boundary. Your database rollback does not undo a CRM task or an email.
The root cause is that the state of the interaction lives only in JVM memory: the conversation so far, which tools ran, and what they returned. When the JVM stops, for a rolling restart, an OOM kill, or a recycled node, that state is gone. There is no record anywhere that says "the policy lookup succeeded, the callback task was created, the model was about to write the final answer."
Why the usual workarounds fall short
Familiar tools each help a little and leave a gap.
@Retryable on the service method. Spring Retry re-invokes the whole method. For an agent call, that means a new model request, a new reasoning path, and every tool executed again. If one of those tools has a side effect, you get the duplicate callback task from the opening scene. It is the wrong granularity for a multi-step interaction.
Longer graceful shutdown. Raising the termination grace period reduces how often in-flight calls get cut off during deploys. It does nothing for crashes or evictions, and it slows every rollout.
Idempotency keys on every tool. Good practice where possible, but it pushes the problem into downstream systems you may not own, and the model still has to be called again.
Logging tool calls. Logs give you a partial audit trail, but logs are not state. Your application cannot resume from a log line, and logs are rarely complete enough for an auditor asking which tool was called, with which arguments, for which customer.
What all of these miss is the same thing: a durable record of each step that the application itself uses to recover.
The general approach: durable execution
The pattern that fixes this is neither AI-specific nor vendor-specific. It is usually called durable execution, or workflow-as-code:
- Model the interaction as a sequence of steps. Each model call and each tool call is a discrete step with an input and an output.
- Persist each step's result before moving on. The record goes to a durable store, not to memory.
- On restart, replay from the record. Completed steps return their saved results instead of executing again. Only the in-flight step runs again.
- Keep the orchestration logic deterministic. Model calls, HTTP requests, and database writes live inside steps, so replay is safe.
- Retry at the step level, not the whole interaction. A timed-out tool call is retried without re-running tools that already succeeded.
With this model, the rolling restart becomes boring. The new pod sees that the policy lookup and callback task already completed and continues from the model's next turn. The task is created once, and the recorded steps double as an audit trail of tool calls.
Implementing it in a Spring AI application
Java teams have two practical routes, and they can coexist.
Option 1: make ChatClient calls durable
If you already use Spring AI's ChatClient, the least invasive route is to make each ChatClient.call() durable without rewriting it. One option here is the Spring AI starter from Diagrid, published as io.diagrid:diagrid-spring-ai-starter. Be clear-eyed about its status: it is in early development (version 0.2.0 at the time of writing), and its APIs are not yet stable. Treat it as something to evaluate in a non-critical service first, not something to drop into a regulated production path next sprint.
With the dependency on the classpath, every Spring-managed ChatClient.call(), along with the tool calls it triggers, runs as a Dapr Workflow. If the JVM dies mid-interaction, the workflow resumes from the last completed step rather than starting over. No application code changes are required for clients built the expected way.
<dependency>
<groupId>io.diagrid</groupId>
<artifactId>diagrid-spring-ai-starter</artifactId>
<version>0.2.0</version>
</dependency>
The phrase "Spring-managed" matters, and it is the gotcha most teams will hit. Durability is applied to clients built from the ChatClient.Builder that Spring injects. So inject the builder through the constructor and build from it:
@Service
public class PolicyAssistant {
private final ChatClient chatClient;
public PolicyAssistant(ChatClient.Builder builder) {
this.chatClient = builder
.defaultSystem("You help call-center staff answer policy coverage questions.")
.build();
}
}
The trap is code that builds a client manually with ChatClient.builder(chatModel) inside a component. That client works and passes tests, but it is silently not durable. If you need a manually built client, add the injectable DurableAdvisor with .defaultAdvisors(durable). Grep for ChatClient.builder( during review and confirm each hit is covered.
Option 2: write workflows directly with Dapr's Java SDK
Some interactions deserve explicit orchestration: a claims triage flow that waits for a human approval, fans out to several checks, or needs compensation if a later step fails. For those, Dapr Workflow has a Java SDK you can call from a Spring Boot service. Dapr Workflow keeps an event-sourced history in a state store, replays after a crash, and does not re-execute completed activities. It supports chaining, fan-out/fan-in, durable timers, external events for approvals, and compensation. The usual rule applies: keep the orchestrator deterministic and put I/O, including model calls, in activities.
Where the runtime lives
Both routes need a Dapr runtime with workflows enabled. You can run a self-hosted Dapr sidecar next to your Spring Boot app, on Kubernetes or VMs, with a state store you already operate, such as PostgreSQL. Alternatively, a managed durable execution platform for AI agents such as Catalyst provides the same model, with deployment options that include Diagrid-hosted, bring-your-own-cloud, self-hosted, and air-gapped environments, which matters for banks and insurers with strict data residency rules. Whichever you pick, run your own failure tests rather than relying on vendor descriptions: kill pods mid-call and verify what actually happens.
A practical rollout checklist
Use this roadmap to bring one Spring AI feature up to your existing reliability bar.
- ☐ Inventory every place your code creates a ChatClient, and note which tools each one can call.
- ☐ Classify each tool as read-only or side-effecting. Side-effecting tools (creating tasks, sending messages, updating records) are your highest risk.
- ☐ Remove @Retryable from methods that wrap whole agent interactions, and move retries to the step level.
- ☐ Add idempotency keys to side-effecting tools where the downstream system supports them, as a second line of defense.
- ☐ Choose a route: durable ChatClient calls for simple assistants, explicit Dapr Workflows for multi-step processes with approvals or compensation.
- ☐ Standardize on the injected ChatClient.Builder, and flag any ChatClient.builder(chatModel) usage in code review.
- ☐ Decide where the runtime and its state store live, and confirm that choice with your security and data-residency owners.
- ☐ Write a chaos test: start an interaction, kill the pod after the first side-effecting tool, and assert the tool ran exactly once and the interaction completed.
- ☐ Export OpenTelemetry traces to your existing observability stack so tool calls are auditable.
- ☐ Run a rolling restart under load in staging before calling it done.
Conclusion
Spring teams do not need to lower their standards to ship AI features. The instincts are right; they just need to be applied at the correct granularity. An agent interaction is a sequence of expensive, non-deterministic, side-effecting steps, and the reliable way to run it is to record each step durably and resume from that record after a failure.
Whether you get there with a starter that wraps ChatClient, with hand-written Dapr Workflows, or with both, the test is the same one the insurer's team should have run before that Tuesday change window: restart the service in the middle of a conversation, and check that the customer gets exactly one callback.
