Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows

Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows Large Language Models are excellent at generating text. But generation alone isn't enough to build truly useful AI applications. Imagine asking an AI assistant: What's the status of my order? A normal LLM can explain how order tracking works. But it cannot magically access your order database. Or suppose you ask: Cancel my order #ORD-10291. The model can tell you how to cancel an order. But it cannot actually cancel anything unless your application gives it the ability to perform that action. This is where tool calling and AI agents come in. Instead of simply generating an answer, an AI application can: Understand the request ↓ Decide what action is required ↓ Select a tool ↓ Execute the tool ↓ Observe the result ↓ Continue reasoning ↓ Generate the final response In this article, we'll explore how to build this architecture using Spring AI. What Is an AI Agent? An AI agent is an application where an LLM can decide what actions need to be performed and use available tools to accomplish a goal. A traditional LLM application looks like: User ↓ Prompt ↓ LLM ↓ Response An agent-based application looks more like: User ↓ LLM ↓ Decision ↓ Tool ↓ Result ↓ LLM ↓ Decision ↓ Another Tool ↓ Result ↓ Final Answer The important difference is: The LLM is no longer limited to generating text. It can interact with the application through controlled capabilities. Tool Calling Tool calling allows an LLM to request the execution of a function exposed by your application. For example, imagine our application provides: getOrderStatus() cancelOrder() getCustomer() createSupportTicket() The user asks: Where is my order? The model might determine that it needs: getOrderStatus() The application executes the function and returns: Order #10291 Status: Shipped Expected delivery: September 10 The LLM can then generate: Your order has been shipped and is expected to arrive on September 10. The LLM didn't directly access the database. Instead: LLM ↓ Tool Request ↓ Application ↓ Database ↓ Tool Result ↓ LLM ↓ Answer This distinction is extremely important for enterprise applications. Why Tool Calling Matters Without tools: LLM ↓ Text With tools: LLM ↓ Tools ├── Database ├── REST APIs ├── Search ├── Payment systems ├── CRM ├── Internal services └── Business workflows This turns the LLM from a text-generation component into an interface for interacting with your application. For example, an AI sales assistant could have: getCustomer() getCustomerOrders() createLead() updateLead() sendEmail() scheduleMeeting() A support agent could have: searchKnowledgeBase() getCustomerAccount() getOrder() createTicket() updateTicket() An internal developer assistant could have: searchDocumentation() searchGitRepository() getBuildStatus() createIssue() The possibilities are much broader than simple question answering. RAG vs Tool Calling At this point, it is useful to distinguish RAG from tool calling. RAG is primarily about retrieving information. Tool calling is about performing actions or retrieving live data through application capabilities. For example: RAG ↓ Retrieve company documentation ↓ Answer question Tool calling: LLM ↓ Call order API ↓ Get live order status ↓ Answer They can also be combined. For example: User ↓ AI Agent ├── RAG → Search company policies │ ├── Tool → Get customer account │ └── Tool → Check order status ↓ LLM ↓ Answer This combination is extremely powerful. Spring AI and Tool Calling Spring AI provides abstractions that make it easier to expose application capabilities to language models. A simplified architecture looks like: Spring Boot │ ├── ChatModel │ ├── Tools │ ├── Advisors │ ├── Chat Memory │ └── Vector Store The application controls which tools are available. The model decides whether a tool is needed. This separation is important. The model should not have unrestricted access to your application. Instead, the application exposes specific capabilities. Creating a Tool Imagine we have an order service. @Service public class OrderService { public OrderStatus getOrderStatus(String orderId) { // Fetch order from database return orderRepository.findStatus(orderId); } } We can expose a controlled method as an AI tool. Conceptually: @Tool( description = "Get the current status of an order" ) public OrderStatus getOrderStatus(String orderId) { return orderService.getOrderStatus(orderId); } The description is important. The model uses the tool description to understand: What does this tool do? When should I use it? What parameters does it require? For example: Tool: getOrderStatus Description: Returns the current shipping and delivery status for a customer order. Input: orderId The model can then determine whether this tool is appropriate. Tool Schema A tool can be thought of as: Tool Name + Description + Input Schema + Execution Logic For example: getOrderStatus( orderId: String ) The model might produce a tool request conceptually like: { "name": "getOrderStatus", "arguments": { "orderId": "ORD-10291" } } The application receives the request and executes the corresponding Java method. The Tool Calling Loop A typical interaction looks like this: User │ │ "What's the status of ORD-10291?" ↓ LLM │ │ Tool Request ↓ getOrderStatus("ORD-10291") │ ↓ Order Service │ ↓ Database │ ↓ Tool Result │ ↓ LLM │ ↓ Final Answer Notice something important. The LLM doesn't execute Java code itself. The application remains responsible for execution. The model only requests the action. Tool Calling with ChatClient Spring AI's ChatClient provides a convenient API for interacting with chat models. Conceptually: ChatClient chatClient; String response = chatClient.prompt() .user("What's the status of order ORD-10291?") .tools(orderTools) .call() .content(); The exact APIs may vary depending on the Spring AI version you're using, but the architecture remains the same: ChatClient ↓ ChatModel ↓ Tool Selection ↓ Tool Execution ↓ Tool Result ↓ Final Response Multiple Tools A real agent usually has more than one tool. For example: CustomerAgentTools ├── getCustomer() ├── getCustomerOrders() ├── getOrderStatus() ├── createSupportTicket() └── updateCustomer() Now consider this question: My order is late. Please check the status and create a support ticket if necessary. The model might determine: 1. getOrderStatus() 2. Analyze result 3. createSupportTicket() 4. Return final response The application executes each requested operation. This is where the concept of an agent starts becoming much more interesting. Agents and Decision Making A simple agent loop can be represented as: ┌───────────────┐ │ User │ └───────┬───────┘ ↓ ┌───────────┐ │ LLM │ └─────┬─────┘ ↓ Need a Tool? / \ No Yes ↓ ↓ Final Answer Tool Call ↓ Tool Execution ↓ Tool Result ↓ LLM ↓ Need Another Tool? The model can repeatedly interact with tools until it has enough information to produce the final response. Tool Calling Is Not Full Autonomy This distinction is important. People often hear: AI Agent and immediately think: Give AI access to everything ↓ Let AI do whatever it wants That is not how production systems should be designed. A production agent should operate inside clear boundaries. For example: Allowed Tools ↓ Authorization ↓ Validation ↓ Execution ↓ Audit The application remains in control. The model should not be trusted with unrestricted capabilities. Tool Security Imagine we expose: @Tool public void deleteCustomer(String customerId) { ... } This is potentially dangerous. An LLM should not automatically receive unrestricted permission to perform destructive operations. Instead, sensitive tools should have additional controls. For example: User ↓ Authentication ↓ Authorization ↓ Agent ↓ Tool Request ↓ Permission Check ↓ Confirmation ↓ Execution For destructive operations, you may require explicit user confirmation. Example: AI: I found customer account C-19291. Deleting this account is irreversible. Do you want me to continue? User: Yes. AI: Executing deletion... The AI should assist with the decision process, not bypass your security model. Tool Permissions A useful production architecture is to classify tools. READ getCustomer() getOrder() searchDocuments() getInvoice() Then: WRITE createTicket() updateCustomer() createLead() And: DESTRUCTIVE deleteCustomer() cancelSubscription() refundPayment() Different permission levels can then be applied. For example: READ → Automatically allowed WRITE → Role-based authorization DESTRUCTIVE → Authorization + confirmation This makes agent behavior much safer. AI Agent Memory Tool calling solves one problem. But another problem appears quickly: What does the agent remember? Consider this conversation: User: My order is late. AI: What's your order number? User: ORD-10291. AI: Let me check it. Now the next message is: Can you create a support ticket for it? The AI needs to understand that: "it" refers to: ORD-10291 This requires conversational context. That's where chat memory becomes important. Chat Memory A simple conversation can be represented as: User: My order is late. Assistant: What's your order number? User: ORD-10291. Assistant: Let me check that order. The application maintains the conversation history. Conceptually: Conversation ID ↓ Chat Memory ↓ Previous Messages ↓ Current Prompt ↓ LLM Spring AI provides abstractions for managing chat memory. Short-Term vs Long-Term Memory It is useful to distinguish two concepts. Short-Term Memory Conversation context. User: My order is late. User: It's order 10291. User: Can you check it? The system remembers the current conversation. Long-Term Memory Persistent information about the user. For example: Customer: Ayush Preferences: Preferred language = English Preferred notification = Email Long-term memory usually requires persistence in a database or another storage system. A production architecture might look like: Conversation ↓ Chat Memory Store ↓ PostgreSQL / Redis The exact storage mechanism depends on the application. Agents + Memory Now we can combine: User ↓ Agent ↓ Memory ↓ LLM ↓ Tools ↓ Tool Results ↓ Memory ↓ LLM ↓ Answer This enables more natural multi-turn interactions. Advisors Another important Spring AI concept is the Advisor. Advisors can intercept and influence the interaction between the application and the model. They can be used for concerns such as: Conversation memory RAG Logging Security Prompt modification Context injection Observability Conceptually: User ↓ ChatClient ↓ Advisor ↓ ChatModel ↓ Advisor ↓ Response This allows cross-cutting AI behavior to be separated from business logic. Combining RAG + Tools + Memory Now things become much more powerful. Imagine an enterprise support agent. It has: RAG ↓ Company documentation Tools: getCustomer() getOrder() createTicket() Memory: Conversation history The architecture becomes: User ↓ AI Agent ↓ ┌───────┼────────┐ ↓ ↓ ↓ RAG Tools Memory ↓ ↓ ↓ Knowledge APIs Conversation │ │ │ └───────┼────────┘ ↓ LLM ↓ Response This is much closer to a production AI application. Example: Customer Support Agent Consider the request: My payment failed for order ORD-19291. Can you check what happened and tell me what I should do? The agent could perform: 1. getOrder("ORD-19291") 2. getPaymentStatus("ORD-19291") 3. searchKnowledgeBase("payment failure") 4. Generate explanation The final answer could be: Your payment attempt failed because the transaction was declined by the payment provider. According to the payment policy, you can retry the payment using another payment method. Would you like me to create a support ticket? The model combined: Live application data + Knowledge base + Conversation context This is significantly more useful than a standalone chatbot. Multi-Step Agent Workflow Agents can also perform multi-step workflows. For example: User: Find my overdue invoices and send reminders. The agent could reason through: getCustomer() ↓ getInvoices() ↓ Filter overdue invoices ↓ sendReminder() ↓ Return summary The workflow becomes: Goal ↓ Plan ↓ Tool ↓ Observe ↓ Next Decision ↓ Tool ↓ Observe ↓ Final Result This pattern is often called an agent loop. Agent Loop A simplified conceptual implementation looks like: while (!completed) { AgentDecision decision = llm.decide(context); if (decision.requiresTool()) { ToolResult result = toolExecutor.execute( decision.toolCall() ); context.add(result); } else { return decision.finalAnswer(); } } In real applications, frameworks handle much of this interaction. But understanding the underlying loop is important. Don't Build Everything as an Agent An important engineering lesson: Not every AI feature needs an agent. If your workflow is deterministic: Validate request ↓ Call API ↓ Save result ↓ Return response you probably don't need an autonomous agent. A normal service workflow may be better. Agents become more useful when: The next step depends on the current result. For example: Check order ↓ If delayed ↓ Check refund policy ↓ If eligible ↓ Ask for confirmation ↓ Create refund request The dynamic decision-making is where agents become valuable. Deterministic Workflow vs Agent Traditional Workflow A → B → C → D Everything is predetermined. Agent Workflow A ↓ LLM decides ├── B ├── C └── D ↓ Observe result ↓ Decide again Agents provide flexibility. Traditional workflows provide predictability. Production systems often use both. Agent Architecture for Enterprise Java A practical Spring Boot architecture might look like: ┌───────────────┐ │ Frontend │ └───────┬───────┘ ↓ ┌───────────────┐ │ Spring Boot │ │ API │ └───────┬───────┘ ↓ ┌────────────┐ │ ChatClient │ └─────┬──────┘ ↓ ┌──────────────┐ │ Agent │ └──────┬───────┘ ↓ ┌────────────┼────────────┐ ↓ ↓ ↓ Memory RAG Tools ↓ ↓ ↓ PostgreSQL pgvector APIs ↓ Microservices This architecture fits naturally into existing Spring Boot applications. Observability Agent systems can become difficult to debug. Imagine an agent performs: Tool 1 Tool 2 Tool 3 Tool 4 and the final response is incorrect. You need to know: What did the model decide? Which tools were selected? What arguments were sent? How long did each tool take? What did each tool return? How many model calls happened? How many tokens were consumed? Therefore, observability is critical. Track: LLM latency Tool latency Retrieval latency Token usage Tool calls Tool failures Model responses Agent iterations Errors Preventing Infinite Agent Loops An agent can potentially continue calling tools indefinitely. For example: LLM ↓ Tool ↓ LLM ↓ Tool ↓ LLM ↓ Tool ↓ ... Production systems should enforce limits. For example: Maximum iterations = 10 Maximum tool calls = 20 Maximum execution time = 30 seconds You should also define clear failure behavior. Agent limit reached ↓ Stop execution ↓ Return safe response ↓ Log failure Tool Validation Never blindly trust model-generated tool arguments. Suppose the model requests: { "orderId": "ORD-999999999" } Your application should still validate: Does the order exist? Does the user own the order? Is the user authorized? Is the order accessible to this tenant? The architecture should be: LLM ↓ Tool Request ↓ Schema Validation ↓ Authorization ↓ Business Validation ↓ Tool Execution The LLM is not your security boundary. Your application is. Multi-Tenant AI Agents This becomes especially important in SaaS applications. Imagine: Tenant A ├── Customers ├── Orders └── Documents Tenant B ├── Customers ├── Orders └── Documents An AI agent must never retrieve Tenant B's information while processing a Tenant A request. Every tool and retrieval operation should carry tenant context. For example: tenant_id user_id roles permissions Then: User ↓ Authentication ↓ Tenant Context ↓ Agent ↓ Tool ↓ Authorization ↓ Tenant-scoped Data The same principle applies to RAG. Vector Search + tenant_id filter should ensure that retrieved documents belong to the correct tenant. AI Agent Guardrails Production agents should have explicit guardrails. Examples: Input validation Output validation Tool authorization Rate limiting Token limits Iteration limits PII protection Audit logging Human approval For high-risk actions: Agent ↓ Tool Request ↓ Risk Evaluation ↓ Human Approval ↓ Execution This creates a human-in-the-loop architecture. Human-in-the-Loop Not every decision should be fully automated. For example: Refund amount < $50 ↓ Automatic Refund amount > $50 ↓ Human approval Or: Create support ticket ↓ Automatic Delete account ↓ Confirmation required This gives us a practical balance: AI Automation + Business Rules + Human Oversight RAG + Tool Calling + Memory At this point, we can combine everything we've discussed. User ↓ Spring Boot ↓ ChatClient ↓ AI Agent ↓ ┌───────────┼───────────┐ ↓ ↓ ↓ Memory RAG Tools ↓ ↓ ↓ PostgreSQL pgvector REST APIs ↓ Business Services ↓ Database This is a strong foundation for enterprise AI applications. Example Enterprise Agent Imagine a sales assistant. The user asks: Show me the latest opportunities for Acme and tell me which ones are likely to close this month. The agent could: 1. getCustomer("Acme") 2. getOpportunities("Acme") 3. retrieve sales documentation 4. analyze opportunity information 5. generate summary Now the user says: Create a follow-up task for the highest priority opportunity. The agent can: 1. Identify opportunity 2. createFollowUpTask() 3. Return task details This is where AI starts becoming an application interface rather than simply a chatbot. A Useful Mental Model Think about the responsibilities this way: LLM = Reasoning + Language RAG = Knowledge Retrieval Tools = Actions + Live Data Memory = Conversation Context Spring Boot = Application + Security + Business Logic Together: LLM + RAG + Tools + Memory + Business Logic = AI Application What Spring AI Gives Java Developers Spring AI provides abstractions that allow Java developers to work with AI capabilities using familiar Spring patterns. Important building blocks include: ChatClient ChatModel EmbeddingModel VectorStore Document Advisors Chat Memory Tools This means an enterprise Java team can integrate AI into an existing Spring Boot architecture instead of creating an entirely separate AI stack. For example: Existing Spring Boot Application ↓ Spring AI Layer ↓ Model + RAG + Tools ↓ Existing Microservices This makes AI integration much more practical for Java teams. Production Architecture A more complete production system might eventually look like: ┌───────────────┐ │ User │ └───────┬───────┘ ↓ API Gateway ↓ Authentication ↓ Spring Boot API ↓ AI Agent ↓ ┌──────────────────────┼──────────────────────┐ ↓ ↓ ↓ Memory RAG Tools ↓ ↓ ↓ PostgreSQL pgvector Microservices ↓ Business Database ↓ LLM Provider ↓ Response And around the entire system: Security Observability Rate Limiting Audit Logging Guardrails Evaluation These are not optional concerns in serious enterprise deployments. Agent vs Chatbot It is useful to understand the difference. Chatbot User ↓ LLM ↓ Answer RAG Chatbot User ↓ Retrieve Knowledge ↓ LLM ↓ Answer Tool-Enabled Assistant User ↓ LLM ↓ Tool ↓ Result ↓ Answer AI Agent User ↓ Agent ↓ Reason ↓ Tool ↓ Observe ↓ Reason ↓ Tool ↓ Observe ↓ Final Answer The complexity increases at every stage. The Agentic AI Stack We can now think about the evolution of an AI application: Level 1 LLM ↓ Text Generation Level 2 LLM + RAG ↓ Knowledge Retrieval Level 3 LLM + Tools ↓ Actions Level 4 LLM + Tools + Memory ↓ Contextual Assistant Level 5 LLM + RAG + Tools + Memory ↓ Agent Level 6 Multiple Agents + Workflows ↓ Agentic System This progression is useful when deciding how much complexity your application actually needs. Final Takeaway The evolution from traditional AI applications to agentic applications can be summarized as: LLM ↓ Generate Text RAG ↓ Retrieve Knowledge Tool Calling ↓ Take Actions Memory ↓ Remember Context Agents ↓ Make Decisions Workflows ↓ Coordinate Multiple Steps Spring AI provides Java developers with abstractions for building many of these capabilities inside the Spring ecosystem. The most important engineering principle is: Let the model decide, but let your application control. The LLM can decide which tool may be useful. Your application should decide whether that tool is actually allowed to execute. That separation gives us a much safer architecture for enterprise AI. What's Next? We've now covered three major capabilities: LLM ↓ Generate RAG ↓ Retrieve Tools ↓ Act But there is another challenge. What happens when a system has: Multiple agents ↓ Multiple tools ↓ Multiple services ↓ Multiple AI models How do these agents communicate? How do we standardize tool discovery? How can an AI agent securely interact with external tools and services? This leads us toward another important concept in modern AI engineering: Model Context Protocol — MCP. In the next article, we'll explore: Building MCP Clients and Tool-Based AI Applications with Spring AI. Key Takeaways Tool calling allows LLMs to interact with application capabilities. The LLM requests tools; your application executes them. Tools can expose APIs, databases, business operations, and services. Chat memory provides conversational context. RAG provides external knowledge. Agents can combine RAG, memory, and tools. Not every workflow requires an agent. Deterministic workflows are often better for predictable business processes. Tool permissions and authorization are critical. Destructive operations should require stronger controls. Multi-tenant applications must enforce tenant isolation at the tool and retrieval layers. Agent loops should have iteration, timeout, and tool-call limits. Observability is essential for debugging agent behavior. Human-in-the-loop approval is useful for high-risk actions. Spring AI provides abstractions that make these patterns accessible to Java and Spring Boot developers. The future of enterprise AI isn't just: LLM → Answer It's increasingly: LLM ↓ Reason ↓ Retrieve ↓ Call Tools ↓ Observe ↓ Act ↓ Remember ↓ Complete the Goal And that's where AI agents with Spring AI become truly interesting.