Policy Monitoring is currently in active development. Reach out to support@craftcx.com for early access.
Why it matters
AI agents can produce helpful, confident-sounding responses while still violating your refund rules, authorization limits, escalation procedures, legal requirements, or internal operating guidance. These violations are difficult to catch through manual spot-checks alone. Policy Monitoring gives your team continuous coverage across every AI-handled conversation—not just a sample—so nothing slips through.How it works
Connect your policy sources
Upload your policy documents (PDFs, Word documents) or connect a Notion workspace. CraftCX supports multiple source documents per policy area.
CraftCX extracts your rules
CraftCX reads your documents and extracts the specific requirements, procedures, and commitments that AI interactions should follow. You can review active and draft rules in the Rules section of the platform.
Every conversation is evaluated
As AI-handled conversations come in, CraftCX checks each one against your active rules. Evaluations happen automatically—no manual review queue needed.
Violations are surfaced with evidence
When a deviation is detected, CraftCX creates a finding that includes a plain-language explanation, the relevant transcript snippets, citations back to the policy source, and a severity and confidence rating.
Repeated violations become incidents
If the same type of violation recurs, or a single violation is severe enough, it escalates into an Incident for your team to investigate and resolve.
Understanding a finding
Each policy finding includes:- Summary & severity
- Evidence
- Why it was flagged
A plain-language description of what happened and why it was flagged. Findings are rated by severity (high, medium, low) and a confidence score, so your team can prioritize the most impactful issues first.
Triaging findings
Your team can act on each finding directly in CraftCX:| Action | When to use it |
|---|---|
| Acknowledge | You’ve reviewed it and are aware of the issue |
| Needs fix | Something in your policy, AI configuration, or process needs to change |
| False positive | The finding was incorrectly flagged |
| Resolved | The underlying issue has been addressed |
Managing your policies
Uploading documents
Go to Policy Docs in the platform to upload PDF or Word documents. CraftCX will parse the documents and extract evaluable rules.Connecting Notion
If your policies live in Notion, you can connect your workspace directly. CraftCX will pull content from the pages you select and keep rules in sync as your documents change.Reviewing extracted rules
After uploading a document, CraftCX generates draft rules from it. Go to Rules to review what was extracted before it goes live. Rules move from draft to active once reviewed.Frequently Asked Questions
Which conversations does Policy Monitoring evaluate?
Which conversations does Policy Monitoring evaluate?
CraftCX evaluates AI-handled conversations that have been ingested through your connected helpdesk integration. Human-only conversations are not included in policy evaluations.
How quickly do findings appear after a conversation ends?
How quickly do findings appear after a conversation ends?
Policy evaluations typically complete within a few minutes of a conversation being resolved. Timing may vary slightly depending on your helpdesk’s resolution status updates.
What if my AI was actually correct, but the policy is ambiguous?
What if my AI was actually correct, but the policy is ambiguous?
Mark the finding as a false positive and note the ambiguity. CraftCX also detects stale, conflicting, or unclear guidance in your policy documents over time, which helps you identify where policies need clarification.
Can I see trends across all findings for a specific policy?
Can I see trends across all findings for a specific policy?
Yes. The Monitor view lets you filter findings by policy rule and see how violations trend over time. Recurring violations for the same rule are a strong signal that either the AI configuration or the policy itself needs attention.
What file formats are supported for policy documents?
What file formats are supported for policy documents?
CraftCX currently supports PDF and Word documents for uploads. You can also connect a Notion workspace to pull policy content directly from pages.
How is this different from AXIS scoring?
How is this different from AXIS scoring?
AXIS scoring measures the overall quality of a conversation—whether it was resolved accurately, efficiently, and with smooth handoffs. Policy Monitoring specifically checks whether AI behavior followed your defined rules and procedures. A conversation can score well on AXIS while still violating a specific policy, or vice versa.
Incidents
See how repeated violations escalate into incidents your team can track and resolve.
Notifications
Get alerted when a policy violation is detected.