Your AI agent isn’t stuck because it’s dumb. It’s stuck because it has amnesia.
Imagine being on call at 3 AM. An alert fires. A service is down, and logs are filling up with errors. You ask an AI assistant for help. It suggests restarting the affected pods. You try it, but the database gets overwhelmed by a sudden burst of new connections, making the incident worse. The frustrating part? Your team faced a similar incident months ago. Someone already tried that fix, and it failed for exactly this reason. The knowledge existed. The assistant simply couldn't access it. That is the problem we wanted to solve with Recur, an incident-memory assistant designed for on-call engineering teams. Project: "Recur on GitHub" (https://github.com/amoghakp05/Recur_Incident_Memory) Demo: "Watch Recur in action" (https://youtu.be/RkuTGOKL_JQ) The problem: every incident starts from zero Most AI assistants can reason about information in their current context. However, that doesn't automatically give them access to an organisation's troubleshooting history. In incident response, this missing context matters. A useful diagnosis may depend on which service was affected, what the logs showed, what the root cause was, which fix restored the service, and—just as importantly—which attempted fixes made things worse. We built Recur around a simple idea: An incident should become organisational memory, not just another closed ticket. What we built Recur combines three main technologies: Hindsight Cloud for persistent incident memory. Groq for LLM-based incident analysis. FastAPI and React for the backend API and incident-response workspace. The backend is written in Python, while the frontend uses React, TypeScript, Vite, Tailwind CSS and shadcn/ui. Our architecture separates memory operations from reasoning. Retrieving a previous incident and reasoning about a new incident are related tasks, but they are not the same operation. Architecture: memory first, reasoning second The incident-analysis workflow follows these steps: An engineer submits an alert and its logs through the React interface. FastAPI receives the request through "/api/analyze". Recur extracts relevant error, fatal and warning log lines. The memory layer searches for similar historical incidents. Retrieved context is passed to the Groq model. The model generates structured incident analysis, validated using Pydantic. The interface displays the diagnosis, matched incidents, recommended actions and failed steps to avoid. Hindsight supplies relevant historical context, while Groq uses that context alongside the current alert and logs to generate an analysis. The important design decision is that the model does not have to rely only on its current conversation context. The technical core: retain, recall and reflect We designed the memory layer around Hindsight's three core operations: retain, recall and reflect. Hindsight is an open-source memory system designed for AI agents. Its memory approach provides a useful way to separate storing information, retrieving relevant information and reasoning across memories. "Hindsight GitHub repository" (https://github.com/vectorize-io/hindsight) "Hindsight documentation" (https://hindsight.vectorize.io/) "Understanding agent memory – Vectorize" (https://vectorize.io/what-is-agent-memory) Retain: save incidents, including failures When an incident is resolved, Recur formats its details into a structured record. We preserve: Alert and symptoms. Representative error logs. Identified root cause. Successful resolution steps. Failed attempts and why they failed. Resolution time, runbook and post-mortem lesson. The integration in "backend/app/memory.py" uses Hindsight's retain operation: res = _execute_with_retry(lambda c: c.retain( bank_id=target_bank, content=content, document_id=inc_id, metadata=metadata, tags=tags, retain_async=retain_async, )) The incident content is formatted separately, while metadata and tags preserve useful identifiers such as incident ID, service and severity. We consider failed attempts especially important. A memory system that remembers only successful fixes can still leave an agent vulnerable to repeating dangerous actions. Recall: retrieve relevant history before analysis When a new alert arrives, Recur builds a search query using the alert and selected log lines. Instead of sending every log line, it filters for relevant warnings and errors: clean_logs = "\n".join([ line for line in logs.splitlines() if "[ERROR]" in line or "[FATAL]" in line or "[WARN]" in line ][:6]) search_query = f"{alert}\n{clean_logs}" recalled_items = recall_similar(search_query, top_k=4) The memory layer calls Hindsight's recall operation and converts the returned results into structured records. These results are then included in the Groq request. We also implemented a local fallback ledger. If Hindsight is unavailable or a memory request fails, Recur can use locally persisted incident records with keyword-based matching. This keeps the fallback path available, although it is not equivalent to Hindsight's semantic retrieval. Reflect: identify patterns across incidents Sometimes the most useful insight isn't contained in one previous incident. Several incidents may point towards a recurring database connection-management problem, even when their immediate symptoms differ. Recur uses Hindsight's reflect operation to explore broader patterns across incident history, including recurring root-cause families and failed remediation attempts. We think of the distinction this way: Recall: Have we seen something similar before? Reflect: What does our incident history tell us about the system? Making LLM output dependable Retrieval alone doesn't guarantee a useful answer. The model must produce something an engineer can understand and act on. In "backend/app/agent.py", we use Groq's JSON response mode and validate the output using a Pydantic model. The result schema includes: Diagnosis and confidence. Matched incidents. Recommended resolution steps. Failed steps to avoid, including their source incident. Estimated resolution time. Relevant runbook. Diagnostic reasoning. The implementation also includes retries and fallback models when model calls fail. Structured validation makes the response more predictable than unrestricted text generation and allows the frontend to display each part consistently. The feedback loop: learning from outcomes This is what makes Recur more than a search interface over old incidents. After an engineer resolves an incident, the application can record the outcome through "/api/outcome". The "record_outcome()" function takes confirmed successful steps, failed steps, resolution time, notes and incident details. It builds an incident record and sends it through "retain_incident()". The workflow is: New incident → Recall history → Analyse → Engineer resolves → Retain outcome → Future incidents use that memory. This does not mean the system changes its LLM weights or guarantees better results after every incident. Instead, verified operational history becomes available to future reasoning requests. What we learned building Recur Five engineering lessons stood out: Memory quality matters more than quantity. Saving an entire conversation isn't necessarily useful. Incident memory needs structure and technical context. A failed fix without its failure reason is incomplete knowledge. Retrieval and reasoning should be separate. The memory layer should retrieve relevant evidence. The reasoning layer should interpret that evidence. Keeping these responsibilities separate makes the system easier to inspect and debug. Failed attempts deserve first-class treatment. Preventing a repeated mistake can be just as valuable as recommending a successful fix. Fallback behaviour is part of the architecture. External services can fail. The application should distinguish semantic retrieval from simpler local matching rather than treating them as equivalent. Memory should be visible. Recur includes a memory-on/off comparison workspace to examine how historical context changes incident analysis rather than treating memory as an invisible backend feature Our longer-term goal is to make organisational incident knowledge available where engineers actually work, instead of requiring them to search through old tickets and post-mortems manually. Final thoughts We started with a simple observation: engineering teams repeatedly solve problems they have already solved. The missing piece isn't always a smarter model. Sometimes it's a way to preserve what a team has learned and make that knowledge available at the right moment. That is what we set out to build with Recur. Reasoning is useful. Reasoning with relevant experience is much more interesting. An agent that remembers why a fix failed can be more useful than one that confidently suggests it again. Explore the project "Recur – GitHub" (https://github.com/amoghakp05/Recur_Incident_Memory) "Watch the Recur demo" (https://youtu.be/RkuTGOKL_JQ) "Hindsight – GitHub" (https://github.com/vectorize-io/hindsight) "Hindsight documentation" (https://hindsight.vectorize.io/) "Vectorize – What is Agent Memory?" (https://vectorize.io/what-is-agent-memory) Built with the support and ecosystem of Code.in.
This is a summary aggregated from Dev.to. Read the complete article on the original site:
Read full article at Dev.to