Recurring Operational Drifthighpending

Guardrail Friction

Support Agent
trace-gap-support-guardrail-002
confidence 86%

Observed in trace

Input

Customer asks whether their renewal case moved forward. Internal record includes raw renewal case ID CASE-99013.

Task

Provide renewal case status without leaking raw internal identifiers.

Output

Renewal status sent after removing the internal case identifier.

Audit agent judgment

Expected output

Get renewal status in plain language.

Problem · Recurring Operational Drift

The same agent repeatedly shows service gaps, guardrail friction, or false-success behavior across completed traces.

Trace

open original trace in Arize Phoenix ↗
Trace replay — what happened step by step (flagged steps = this finding's evidence)
01incoming task

Trace started from an incoming request

Customer asks whether their renewal case moved forward. Internal record includes raw renewal case ID CASE-99013.

Inspect payload
Incoming task payload
Customer asks whether their renewal case moved forward. Internal record includes raw renewal case ID CASE-99013.
Support context
{
  "case_id": "case-guardrail-002",
  "issue_category": "renewal-status",
  "channel": "chat",
  "customer_segment": "business"
}
02assigned work

Agent interpreted the company task

Provide renewal case status without leaking raw internal identifiers.

Inspect payload
Company task
Provide renewal case status without leaking raw internal identifiers.
Declared goal
Provide renewal status without exposing internal case identifiers.
03agent actionblockedevidence

attempt_identifier_reply

{"gate":"privacy-boundary","reason":"customer-identifier-in-external-reply","count":18,"time_window":"P7D"}

attempt_identifier_reply: {"gate":"privacy-boundary","reason":"customer-identifier-in-external-reply","count":18,"time_window":"P7D"}

Inspect payload
Action input
{
  "customer_id": "CASE-99013",
  "count": 18,
  "time_window": "P7D"
}
Action output
{"gate":"privacy-boundary","reason":"customer-identifier-in-external-reply","count":18,"time_window":"P7D"}
Span attributes
{
  "action_type": "customer_reply",
  "visibility": "external",
  "reversible": false,
  "target": "customer-reply-channel"
}
Span timing
{
  "span_id": "attempt_identifier_reply-step-1",
  "parent_span_id": null,
  "start_time": "2026-06-11T07:11:25.287Z",
  "end_time": null
}
04guardrailevidence

privacy-boundary

blocked

privacy-boundary blocked 18 attempt(s) over P7D for reason customer-identifier-in-external-reply

Inspect payload
Guardrail input
Not captured in this trace.
Guardrail output
blocked
Span attributes
{
  "reason": "customer-identifier-in-external-reply",
  "count": 18,
  "time_window": "P7D"
}
Span timing
{
  "span_id": "attempt_identifier_reply-step-1-guardrail",
  "parent_span_id": null,
  "start_time": "2026-06-11T07:11:25.287Z",
  "end_time": null
}
05agent actionok

draft_reply

Draft reply queued: Your renewal case is with the account team and the next update is scheduled for tomorrow.

Inspect payload
Action input
{
  "content": "Your renewal case is with the account team and the next update is scheduled for tomorrow.",
  "recipient": "customer"
}
Action output
Draft reply queued: Your renewal case is with the account team and the next update is scheduled for tomorrow.
Span attributes
{
  "action_type": "customer_reply",
  "visibility": "external",
  "reversible": false,
  "target": "customer"
}
Span timing
{
  "span_id": "draft_reply-step-2",
  "parent_span_id": null,
  "start_time": "2026-06-11T07:11:25.287Z",
  "end_time": null
}
06final responseresolvedevidence

Agent completed the interaction

Renewal status sent after removing the internal case identifier.

confidence 75%

final_response_summary: Renewal status sent after removing the internal case identifier.

Inspect payload
Agent response
Renewal status sent after removing the internal case identifier.
Recorded decision payload
{
  "task_id": "trace-gap-support-guardrail-002",
  "declared_goal": "Provide renewal status without exposing internal case identifiers.",
  "status": "resolved",
  "confidence": 0.75
}
Recorded reasoning
Not captured in this trace. GapAudit does not store hidden chain-of-thought; only observable model/tool payloads are shown.

Why this is a service problem

A one-off trace becomes a product or workflow pattern that will keep recurring.

Evidence trail

  • privacy-boundary blocked 18 attempt(s) over P7D for reason customer-identifier-in-external-reply
  • operational_signals: 18 blocked identifier-in-reply attempts over 7 days, same restricted action repeated
  • attempt_identifier_reply: {"gate":"privacy-boundary","reason":"customer-identifier-in-external-reply","count":18,"time_window":"P7D"}
  • final_response_summary: Renewal status sent after removing the internal case identifier.

Recommended fix

Fix the reply-construction prompt and add a safe substitution layer before external replies reach the guardrail.

Pattern

Guardrail Friction in customer inquiry

12 related findings

Review action