Observability, security and control
- date
- category
- AI Agents
- also in
- Observability · Application Security
- reading
- 3 min / 546 words
This is one of the biggest steps compared to a single agent.
When an agent runs in one team, you can still look at its logs and check by hand what it did.
With dozens of agents the organization needs a proper observability layer.
Not just:
HTTP 200
CPU 30%
but also:
what task did it get?
which model did it use?
which RAG did it use?
which documents did it get back?
which tools did it call?
with what arguments?
what did the tool return?
how many tokens did it use?
how long did the task take?
what did it send outside?
what ended up in the final output?
The trace of one task
For PAY-123 we can see:
PAY-123
|
v
jira.get_issue
|
v
git clone
|
v
model call
|
v
engineering_docs.search
|
v
model call
|
v
edit files
|
v
run tests
|
v
github.create_pull_request
Each step can be a span in a trace.
It starts to look like the distributed tracing known from microservices.
What we measure operationally
Runtime
- request count,
- latency,
- failures,
- retries,
- CPU/memory.
Model
- model calls,
- token usage,
- latency,
- cost.
Tools
- which tool was used,
- how many times,
- with what result,
- where errors occur.
Workflow
- task duration,
- number of steps,
- how many times the agent went back to the model,
- where it most often stalls.
Analyzing inputs and outputs
With agents it also matters what flows through the system.
We can look for:
- prompt injection,
- attempts to extract the system prompt,
- secrets in prompts,
- API keys in tool arguments,
- personal data,
- confidential data in outputs,
- unauthorized information coming back from RAG.
Example:
tool output
contains AWS_SECRET_KEY
|
v
security control
|
v
block / redact / alert
Or:
retrieved document
contains malicious instruction:
"ignore previous instructions..."
|
v
prompt injection detection
|
v
block / flag
Guardrails and policies
Not everything should depend on the model's decision.
We can have a deterministic policy:
Implementer Agent
MAY:
- read repo
- create branch
- create PR
MUST NOT:
- merge PR
- delete repo
- deploy production
The tool gateway can reject a disallowed action even when the model tries to perform it.
Where we configure observability
Agent repo
Adds telemetry context:
agent = implementer
issue = PAY-123
repo = payments-service
team = payments
and custom spans where needed.
Platform
Provides:
- trace backend,
- logs,
- dashboards,
- alerts,
- retention,
- access to telemetry,
- security filters,
- redaction.
Tool gateway / guardrails
Controls input/output at the boundary of tool calls.
Tools on the market
| Area | Azure | AWS | GCP | Independent |
|---|---|---|---|---|
| Agent tracing | Foundry Tracing + Application Insights | AgentCore Observability + CloudWatch | Cloud Trace + Agent Engine telemetry | OpenTelemetry, Langfuse, LangSmith |
| Runtime metrics | Azure Monitor / App Insights | CloudWatch | Cloud Monitoring | Grafana / OTEL backend |
| Prompt / output safety | Foundry Guardrails, Prompt Shields | Bedrock Guardrails / AgentCore Policy | Vertex AI safety controls (Model Armor) | your own filters |
| Sensitive data | redaction + protected trace data / Azure security stack | sensitive information guardrails | Google Cloud DLP / security controls | custom DLP |
| Tool policy | Foundry identity/policies around tools | AgentCore Policy + Gateway | IAM / Agent Gateway policies | your own policy layer |
An important detail: traces are data too
A trace can contain:
- user input,
- model output,
- tool arguments,
- tool results.
So observability itself can become a source of leaks.
That is why the organization has to control:
- whether we store full prompt content,
- what we redact,
- who has access to traces,
- how long we keep them.
Observability is not an add-on for the end.
With agents it becomes part of the control mechanism.