1. Issue: Insufficient Verification of Prediction Accuracy

A company providing cloud services has a system that ensures all employees can understand the usage status of each contracted company. This widespread access to customer usage data promotes transparency and likely supports better decision-making across the organization.

This company has a robust system in place for sharing customer usage data company-wide. Every week, Looker Studio graphs detailing each contracted company’s usage are automatically posted to OpenChat. What’s more, these posts include a multimodal generative AI’s churn rate risk assessment, rated on a scale of A to E.

This system helps teams like Customer Success prioritize checking high-risk customers’ usage to devise churn rate reduction strategies. Similarly, the Field Sales team uses it to propose additional services to loyal, low-risk clients.

However, a critical issue remains: the AI’s risk assessment accuracy isn’t sufficiently verified. There’s a perceived misjudgment rate of about 30%. To prepare for future business expansion, it’s crucial to gradually establish a framework to improve the AI’s prediction accuracy.

2. Solution: Diligent Data Labeling

The process owner has decided on a clear initial strategy: human data labeling will first be applied to high-risk (D-E) assessments.

To improve the validation of AI-driven risk assessments, this company has refined its workflow. Here’s a breakdown of the key changes:

New Workflow Adjustments:
– Added an OR Gateway for Branching: High-risk assessment reports are now concurrently routed to multiple paths.
– Automated Subject Line Modification: Reports automatically receive a “Risk_” label in their subject line.
– Introduced Human Review Step: A new human task allows for “agree” (Risk) or “disagree” (no risk) judgments on the AI’s high-risk assessment.
– Automated Subject Line Modification for Disagreements: Reports flagged as “disagree” automatically get a “Wolf_Risk_” label (indicating a false alarm or “wolf in sheep’s clothing” risk).

To these enhancements, every high-risk assessment report now carries either a “Risk_” or “Wolf_Risk_” label. This critical improvement empowers all employees to validate the generative AI’s prediction accuracy (and its misjudgment rate), fostering transparency and accountability in the AI’s performance.

Analyzing and labeling AI errors improves prediction accuracy while enabling continuous prompt optimization.

* Data labeling is the essential process where human

3. Customers Case Study

4. Other Business Applications

We will enable AI-based reviews for the following tasks:

5. Related Posts