Audit - AI Trust & Observability

The Audit feature provides an independent, objective assessment of your AI customer support tooling. It helps you verify billing accuracy, understand AI performance, and ensure you’re being charged fairly for AI-powered resolutions.

​

Key Benefits

​

Performance Transparency

Get clear visibility into how your AI tools are performing. See resolution rates, AI involvement levels, and AXIS scores to understand where your AI excels and where it needs improvement.

​

Billing Dispute Support

When an AI vendor charges for a resolution, use CraftCX Audit as an independent and unbiased source of data to help assess whether that charge is justified. Export detailed audit reports to support billing disputes or internal analysis.

​

Operational Insights

Understand the true impact of your AI support tools. Identify patterns in resolution failures, track AI performance trends, and make data-driven decisions about your support operations.

​

How Audit Works

The Audit feature analyzes every conversation that has been evaluated by CraftCX, providing an independent assessment of:

​

Using the Audit Dashboard

The Audit dashboard provides a comprehensive view of your AI support performance with filtering, metrics, and detailed conversation analysis.

​

Dashboard Features

Metrics Overview

• Total audited conversations
• Resolution rate breakdown (resolved, partial, not resolved)
• AI involvement distribution
• Average AXIS scores

Filtering Options

• Date Range - Filter by conversation date
• Resolution Status - Filter by resolved, partial, or not resolved
• AI Involvement - Filter by AI only, AI assisted, or no AI

Conversation Details

• Click any conversation to view detailed audit information
• See full conversation transcript
• Review AXIS score breakdowns
• Access resolution notes and explanations

​

Exporting Audit Data

You can export audit data in two ways: Dashboard Export

1. Set your desired filters (date range, resolution status, AI involvement)
2. Click the “Export” button in the filter bar
3. The export will be processed asynchronously
4. Download the CSV file when ready

API Export Use the Audit API endpoints to programmatically export audit data for integration with your own systems or automated reporting.

​

Understanding Audit Data

​

Resolution Status

The audit independently determines resolution status based on conversation analysis:

• Resolved - Customer’s issue was fully addressed and resolved
• Partial - Some progress was made but the issue wasn’t fully resolved
• Not Resolved - Issue was not resolved, escalated to human support, or marked as failed

​

AI Involvement Classification

AI involvement is determined by analyzing the conversation participants:

• AI Only - All messages were from AI/bot participants
• AI Assisted - Mix of AI and human messages, indicating collaboration
• No AI - Only human agents participated in the conversation

​

Export File Format

Audit exports are provided as CSV files with the following columns:

• ticket_id - Internal CraftCX ticket identifier
• helpdesk_id - External helpdesk ticket ID (if available)
• subject - Conversation subject line
• channel - Support channel (Front, Zendesk, Intercom, etc.)
• resolution_status - Audit-determined resolution status
• ai_involvement - Level of AI participation
• axis_score - Overall AXIS score (average of RA, IE, HS)
• ra - Resolution Accuracy score
• ie - Interaction Effort score
• hs - Handoff Smoothness score
• resolution_note - Any resolution notes or explanations
• audited_at - Timestamp when the audit was performed

​

Use Cases

​

Billing Verification

When an AI vendor charges for a resolution:

1. Export audit data for the relevant time period
2. Filter by the specific conversation or date range
3. Review the independent resolution status assessment
4. Use audit data to verify or dispute billing charges

​

Performance Analysis

To understand AI performance:

1. Filter by AI involvement type to see AI-only vs. AI-assisted conversations
2. Review AXIS scores to identify quality issues
3. Analyze resolution rates to understand success patterns
4. Export data for deeper analysis in spreadsheets or BI tools

​

Operational Improvement

To improve your support operations:

1. Filter for “not resolved” conversations to identify failure patterns
2. Review low AXIS scores to understand quality issues
3. Analyze AI involvement to optimize handoff processes
4. Track trends over time to measure improvement

​

Frequently Asked Questions

How far back can I audit conversations?

You can audit any conversation that has been evaluated by CraftCX. The default view shows the last 30 days, but you can adjust the date range to view older conversations. Export date ranges are limited to 367 days per export.

How quickly is audit data available?

Audit data is available as soon as a conversation has been evaluated by CraftCX. Most evaluations complete within minutes of conversation resolution, though timing can vary based on your helpdesk’s resolution status updates.

Can I use audit data to dispute billing?

Yes. The Audit feature provides independent verification of resolution status that you can use to verify or dispute billing charges from AI vendors. Export the relevant audit data and use it as evidence in billing discussions.

What's the difference between resolution status and AI involvement?

Resolution status indicates whether the customer’s issue was resolved. AI involvement indicates how much the AI participated in the conversation. A conversation can be resolved with or without AI, and AI can be involved even if the issue wasn’t resolved.

How do I export audit data via API?

Use the Audit API endpoints to programmatically create exports. See the API documentation for details on authentication and request formats. Exports are processed asynchronously and you can check status via the API.

Can I filter exports by specific criteria?

Yes. When creating an export (via dashboard or API), you can filter by date range, resolution status, and AI involvement level. This allows you to create targeted exports for specific analysis needs.

​

Next Steps

Ready to start auditing your AI support tooling? Begin by:

1. Reviewing your audit dashboard - See your current AI performance metrics
2. Filtering by resolution status - Identify conversations that weren’t truly resolved
3. Analyzing AI involvement - Understand how much your AI is contributing
4. Exporting audit data - Create reports for billing verification or analysis
5. Using API exports - Integrate audit data into your own systems

The Audit feature gives you the independent verification and transparency needed to ensure fair billing and optimize your AI support operations.

​

API Access

The Audit feature provides both dashboard and API access to audit data.

​

Dashboard Access

The Audit dashboard is available in-app at /audits and provides a full-featured interface for viewing audit data, filtering conversations, and exporting reports.

​

Public API

Audit exports are available via the public API at /v1/evaluations/exports. The API allows you to programmatically create and manage audit export jobs. For complete API documentation including request/response schemas, authentication details, and code examples, see the API Reference documentation. The evaluation exports endpoints are documented in the OpenAPI specification and will be automatically updated when you run the OpenAPI documentation build tool.