When the Logs Are Not Enough

As part of Spectric's Summer Intern Program, the team developed concepts necessary to evaluate the viability of an AI SRE agent to detect, investigate, and remediate system anomalies in realistic settings. Further, the experimental approach required the ability to induce controllable faults so that experiments could be repeated consistently. This project began by building something for the agent to troubleshoot that supported deliberately introduced realistic failures. This was then augmented with log collection in OpenSearch, custom MCP services, and development of a custom agent code to create an AI SRE that inspect the system, gather better evidence, and identify problems that could not be resolved from logs analysis alone.
The following blog post is from work performed during Spectric’s Summer Intern Program. For more information on our Intern Program please reach out to info@spectric.com.
#SpectricLabs#ROADTRIP #AgenticAI #MCP #OpenSearch #Kubernetes #SoftwareEngineering #Troubleshooting #LLMOps #Observability
ROADTRIP
Work started with the creation of the ROADTRIP, a synthetic road-trip planning platform composed of independently deployed services running in Kubernetes. A user submitted a starting point and destination. A route-planning service generated candidate routes, prediction services evaluated conditions such as traffic, weather, and road hazards, and a scoring component combined those results into ranked recommendations.
The trip data and dependencies were artificial by design. The distributed architecture, service interactions, logging pipeline, and failure modes were intended to resemble the kinds of application behavior engineers encounter in production-style systems.
Every service generated structured logs covering requests, errors, latency, and API interactions. Fluentd forwarded those logs into OpenSearch, creating a common source for dashboards, queries, and AI-assisted analysis. The team could also simulate malfunctions such as connection errors, timeouts, and certificate expirations. ROADTRIP therefore provided both the working application and the controlled failure environment needed to develop and test an AI troubleshooting workflow.
The First Version Could Read the Logs
The team built an AI agent to investigate the ROADTRIP application. A user or scheduled job could ask about system status, recent failures, or other observable behavior. The agent used an OpenSearch MCP integration to retrieve the relevant logs and return evidence to the primary reasoning workflow.
That gave the agent a useful starting point. It could search across services, identify suspicious events, summarize what happened, and propose a likely cause.
It also exposed the central limitation of log-only troubleshooting: the initial evidence was not always enough.
A log entry might show that a request failed without showing the condition that caused the failure. A service might appear slow without enough runtime information to distinguish a local problem from a downstream dependency. In those cases, searching the same logs again did not advance the investigation. An engineer would normally check service status, increase logging detail, reproduce the behavior, or run another diagnostic test.
The question became straightforward: could the agent take those next steps through controlled interfaces instead of stopping at a recommendation?
The Team Built the Missing Diagnostic Layer
To extend the agent beyond OpenSearch, the team built a custom Model Context Protocol server using FastMCP. The server ran in the Kubernetes cluster alongside the ROADTRIP application and the agent. It communicated with the agent over HTTP and hosted project-specific diagnostic functions implemented in Python.
These functions became the agent's tools. They allowed the agent to retrieve service status and operational metrics, inspect conditions such as high latency, change the logging verbosity of a selected service, and run tests that produced additional application activity and logs.
The MCP server provided a deliberate boundary between the model and the application. The agent did not receive unrestricted control of Kubernetes or direct access to arbitrary code. Developers defined each available action, its inputs, the code that executed, and the information returned to the agent.
Writing the tools was only part of the work. The team also had to describe each tool precisely. The agent could see a tool's name, description, parameters, and output structure, but it could not inspect the code behind the function. The description therefore had to explain what the tool did, when it should be used, and what it did not provide. Those descriptions directly influenced whether the agent chose the correct action during an investigation.
A Failure Became an Investigation
With the custom MCP server connected, an investigation could move beyond a single pass through the logs.
The agent first queried OpenSearch for concrete evidence from the ROADTRIP services. If that evidence pointed to a service but did not explain the failure, the agent could call a status tool to retrieve current metadata and operational metrics. If the available logging was too coarse, it could call the SetLogLevel tool to increase the service's logging verbosity at runtime.
Changing the log level did not create missing historical information. It changed how much diagnostic detail the service emitted from that point forward. The agent could then use a diagnostic or validation tool to generate new requests, retrieve the resulting logs through OpenSearch, and inspect the additional detail. When the investigation was complete, the logging level could be restored.
The resulting sequence looked more like an engineering workflow than a chatbot exchange:
QUERY LOGS > IDENTIFY AN EVIDENCE GAP > CHECK THE SERVICE > INCREASE LOG DETAIL > GENERATE NEW ACTIVITY > ANALYZE THE NEW EVIDENCE |
This was the project's most important result. The agent successfully used custom tools to find issues that it could not have identified through the original log evidence alone.
The accomplishment was not that an LLM could explain logs in plain language. It was that the team connected reasoning, log retrieval, application-specific diagnostics, and new evidence collection into one controlled loop.
The Agent Also Needed to Understand the Project
Tool access addressed only one part of the problem. The agent also needed the right context.
Troubleshooting a distributed application depends on more than runtime data. Engineers use architecture, service relationships, project terminology, previous findings, and knowledge accumulated during earlier investigations. Without that information, the agent could retrieve logs and invoke tools while still producing generic or poorly grounded analysis.
The team built the agent with the Strands Agents SDK in a FastAPI application deployed to Kubernetes. Persistent context was stored outside the agent in PostgreSQL. The stored context included conversation history, user configuration, durable memories, and project-specific information such as system architecture and troubleshooting notes.
The application used pgvector and sentence-transformer embeddings to retrieve relevant memories by semantic similarity. Before an investigation began, the application resolved the user, project, and conversation, retrieved the appropriate context, and injected it into a newly created agent. The agent then entered its tool-use loop with the logs, diagnostic capabilities, and project knowledge needed for the request.
A Stateless Design Kept Projects Separate
The team also designed the application to support multiple users and projects through the same deployed service.
Instead of keeping one long-running agent with accumulating internal state, the application created a new agent for each request. It loaded the context associated with that user and project, ran the investigation, returned the findings, and destroyed the agent instance. Persistent information remained in PostgreSQL rather than inside the temporary agent.
This separated the application lifecycle from the agent lifecycle. It also allowed project knowledge discovered by one user to be stored within the appropriate project scope and retrieved during later requests without intentionally mixing unrelated project contexts.
Conclusion
By the end of the project, a complete demonstration environment for AI-assisted observability was built:
A distributed ROADTRIP application with route planning, prediction, scoring, and independently deployed Kubernetes services.
Structured logs and metrics forwarded through Fluentd into OpenSearch.
Realistic, controllable failure conditions represented in the application logs.
An AI agent that could query OpenSearch and return evidence-backed findings.
A custom FastMCP server that exposed project-specific diagnostic tools over HTTP.
Tools for checking service status, examining latency-related behavior, changing logging granularity, and running tests that generated new evidence.
A stateless, multi-user agent application with user and project context stored in PostgreSQL and retrieved through semantic search.
No single component was the project. ROADTRIP created the system and its failure conditions. OpenSearch made the logs available. The agent reasoned over the evidence. The custom MCP server gave the agent controlled ways to interact with the application. The memory architecture supplied the project context required to interpret what it found.
Together, those pieces demonstrated a more useful model for AI-assisted troubleshooting. When the initial logs were inadequate, the agent did not have to force a conclusion or hand the problem back immediately. It could recognize the gap, select an approved action, collect better evidence, and continue the investigation.
That is the difference between an AI assistant that talks about a problem and an AI agent that can begin helping an engineer investigate it.
For more information about Spectric’s internship program, contact info@spectric.com.
Spectric Labs Github: https://github.com/spectriclabs
For more information/questions contact info@spectric.com





Comments