Incident investigation with an AI agentSentry and Grafana through MCP
How to investigate an incident with an AI agent: connect Sentry and Grafana through MCP, have it match errors, metrics and release time, limit permissions and verify the hypothesis.
During an incident a person hops between Sentry, Grafana and chat and stitches the picture together themselves. An agent with access to both systems gathers errors, metrics and event times into one draft, while a person checks the hypothesis and decides what to do. Below: how it goes, how to connect Sentry and Grafana, which permissions to give. The basics are in the MCP for an AI agent article.
How the investigation goes
Takeaway
In this exampleThe agent reads errors in Sentry and charts in Grafana, and a person gets one draft of the investigation.
In generalSentry and Grafana connect through MCP as read-only, so the agent changes nothing without your permission. A person checks the hypothesis.
What to doList the systems you searched for the cause in your last incident.
How to connect Sentry and Grafana
Tell the agent
Connect Sentry and Grafana as read-only. Find new errors for the payment service in Sentry over the last hour, find in Grafana when the share of 5xx responses grew, and match it with the release time. Propose a hypothesis and name the source of every fact.
This is the official server grafana/mcp-grafana. The token comes from a service account with minimal rights, and the --disable-write flag forbids changes.
Takeaway
In this exampleThe agent connects both systems, finds the errors and the chart, matches them with the release and names the sources.
In generalBoth systems connect with a command, and read access keeps the investigation safe.
What to doDelegate the connection to the agent and compare its output with what you see in Sentry and Grafana yourself.
What the agent matches
Takeaway
In this exampleThe release, error spike and alert go on one timeline, and you can see what happened first.
In generalA time match gives a hypothesis and does not prove the cause: checking stays with a person.
What to doAsk for a link to a chart or event with every conclusion.
Permissions and risks
TokenA Grafana service account with minimal rights; the value lives in an environment variable.
Read-onlyThe --disable-write flag forbids changes, including creating incidents.
Third-party serverConnect only servers you trust: servers that read external text open the door to prompt injection.
Takeaway
In this exampleThe agent reads errors and charts, and may not change anything in the systems.
In generalPermissions are given per source, and the widest ones stay closed.
What to doCreate a read-only service account and connect Grafana with the --disable-write flag.
Where to start
Takeaway
In this exampleThree steps: Sentry, Grafana, a test on a past incident.
In generalYou know the past incident, so you can see where the agent found the cause and where it erred.
What to doTake a closed incident and ask the agent to find the cause through Sentry and Grafana.