Human labeling of high-risk AI outputs improves prediction accuracy, reduces misjudgment, and enables continuous AI optimization.
1. Issue: Insufficient Verification of Prediction Accuracy
A company providing cloud services has a system that ensures all employees can understand the usage status of each contracted company. This widespread access to customer usage data promotes transparency and likely supports better decision-making across the organization.
This company has a robust system in place for sharing customer usage data company-wide. Every week, Looker Studio graphs detailing each contracted company’s usage are automatically posted to OpenChat. What’s more, these posts include a multimodal generative AI’s churn rate risk assessment, rated on a scale of A to E.
This system helps teams like Customer Success prioritize checking high-risk customers’ usage to devise churn rate reduction strategies. Similarly, the Field Sales team uses it to propose additional services to loyal, low-risk clients.
However, a critical issue remains: the AI’s risk assessment accuracy isn’t sufficiently verified. There’s a perceived misjudgment rate of about 30%. To prepare for future business expansion, it’s crucial to gradually establish a framework to improve the AI’s prediction accuracy.
2. Solution: Diligent Data Labeling
The process owner has decided on a clear initial strategy: human data labeling will first be applied to high-risk (D-E) assessments.
* “Data labeling” is the process where humans identify and add information to various forms of data. For example, this could involve determining if a photo contains a horse, if a video includes footage of a fire, or if a spot on an X-ray image is a tumor. This labeled data is indispensable for training artificial intelligence models.
To improve the validation of AI-driven risk assessments, this company has refined its workflow. Here’s a breakdown of the key changes:
New Workflow Adjustments: – Added an OR Gateway for Branching: High-risk assessment reports are now concurrently routed to multiple paths. – Automated Subject Line Modification: Reports automatically receive a “Risk_” label in their subject line. – Introduced Human Review Step: A new human task allows for “agree” (Risk) or “disagree” (no risk) judgments on the AI’s high-risk assessment. – Automated Subject Line Modification for Disagreements: Reports flagged as “disagree” automatically get a “Wolf_Risk_” label (indicating a false alarm or “wolf in sheep’s clothing” risk).
To these enhancements, every high-risk assessment report now carries either a “Risk_” or “Wolf_Risk_” label. This critical improvement empowers all employees to validate the generative AI’s prediction accuracy (and its misjudgment rate), fostering transparency and accountability in the AI’s performance.
Analyzing and labeling AI errors improves prediction accuracy while enabling continuous prompt optimization.
* Data labeling is the essential process where human
AI randomly assigns inquiries to balance workload and improve team morale.
1. Issue: Uneven Task Assignment
An IT company uses Augmented Reality (AR) technology to deliver interactive services for education and training. To handle customer inquiries related to their cloud services, the company has a dedicated team made up of one leader and five members. Customers submit their inquiries through a web form, and the team responds via email.
However, the current task assignment system for inquiry responses has become uneven, leading to a sense of unfairness within the team. Members who are assigned a heavier workload complain of being overburdened, while those with fewer assignments express anxiety about not being trusted. This imbalance is creating a negative impact on team morale and overall efficiency.
This sense of unfairness can lead to serious team operational problems. It risks lowering team members’ motivation, decreasing productivity, and increasing the risk of staff turnover. Therefore, it’s urgent to implement measures to address and eliminate this unfairness.
2. Solution: AI-Powered Random Assignment
The team leader, the process owner, realized they were unconsciously assigning a disproportionate number of tasks to veteran members who appeared more experienced and available.
To address this, they’ve decided to eliminate the manual “1. Assign Responder” step, previously handled by the leader. In its place, a new, AI-driven process has been introduced.
Under this new process, “x1. Select Responder by AI,” the AI will randomly choose one email address from the five team members. The member associated with the selected email address will then be automatically set as the responder for the “2. Draft Response” step in the subsequent “x2. Set Responder” stage.
Before
View details of the workflow diagram
s1. Inquiry Reception
The customer initiates an inquiry by filling out a web form, providing their email address and the details of their question.
1. Assign Responder
The team leader decides which member will be responsible for drafting the response to the inquiry.
2. Draft Response
The member chosen in the “Assign Responder” step creates the draft response to the customer’s inquiry.
3. Review
The drafted response from the “Draft Response” step is reviewed. This likely involves checking for accuracy, completeness, tone, and adherence to company guidelines.
2′. Rework Response
If issues are identified during the “Review” step, the response is sent back for revisions based on the feedback.
m1. Send Response
Once the response is finalized (after review and any necessary rework), an email containing the answer is sent to the email address the customer provided in the inquiry form.
After
View details of the workflow diagram
s1. Inquiry Reception
The customer initiates an inquiry by filling out a web form, providing their email address and the details of their question.
x1. Select Responder by AI
From the five team members’ email addresses, the AI randomly selects one. This is the core of the new fairness initiative, removing manual bias from the assignment.
x2. Set Responder
The member corresponding to the email address chosen in “x1. Select Responder by AI” is automatically set as the processor for the “2. Draft Response” step. This ensures a seamless transition from AI selection to task execution.
2. Draft Response
The member chosen in the “Assign Responder” step creates the draft response to the customer’s inquiry.
3. Review
The drafted response from the “Draft Response” step is reviewed. This likely involves checking for accuracy, completeness, tone, and adherence to company guidelines.
2′. Rework Response
If issues are identified during the “Review” step, the response is sent back for revisions based on the feedback.
m1. Send Response
Once the response is finalized (after review and any necessary rework), an email containing the answer is sent to the email address the customer provided in the inquiry form.
Create opportunities for reporters to self-identify discrepancies in their past timecard data.
1. Issue: Mistakes in Timecard Entries
SharaShara Systems is a company that specializes in custom system development for businesses. Employee payroll is calculated based on attendance data. If there are any discrepancies in this data, it directly leads to errors in salary payments. Therefore, the administration department rigorously checks the content to ensure accuracy.
However, as SharaShara Systems is in a growth phase, the number of employees has increased, leading to a rapid surge in the volume of attendance reports that need checking. As a result, the number of errors discovered is also trending upwards. Furthermore, with active new hiring, both the quantity of attendance reports and the incidence of errors are expected to continue rising in the future.
Overlooking attendance discrepancies significantly increases the risk of incorrect wage payments, which can severely damage a company’s credibility and brand value. Therefore, it’s crucial to implement concrete measures immediately to mitigate this risk.
2. Solution: Automatically extract reports from the past 14 days
The process owner determined it was crucial for employees to identify attendance reporting errors themselves at an early stage. To achieve this, a step was added to the workflow, automatically extracting attendance report data from the past 14 days before employees submit their clock-in times.
Now, every day, when employees access the screen to report their clock-in time, the past 14 days of their attendance data will be displayed.
Before
View details of the workflow diagram
s1. Weekday 7:00 AM (Timer Start Event)
The flow automatically starts for all employees every weekday at 7:00 AM.
1. Report Clock-In Time
Employees enter their clock-in time when they begin work.
x1. “Working” Status
The “Attendance Status” data field is set to “Working.”
2. Report Clock-Out Time
Employees enter their break time and clock-out time when they finish work. Their work hours are automatically calculated.
x2. “Work Finished” Status
The “Attendance Status” data field is set to “Work Finished.”
x4. Clock-In/Out Data AI Evaluation
AI evaluates the clock-in/out data and outputs the evaluation results.
3. Confirm Work Hours
The employee’s supervisor reviews the employee’s reported clock-in time, break time, clock-out time, and work hours.
They can also view the AI-generated evaluation results from “x4. Clock-In/Out Data AI Evaluation.”
2x. Handle Revisions
Employees review the reason for the revision and correct their clock-in time, break time, and clock-out time.
x3. “Leave” Status
The “Attendance Status” data field is set to “Leave.”
After
View details of the workflow diagram
s1. Weekday 7:00 AM (Timer Start Event)
The flow automatically starts for all employees every weekday at 7:00 AM.
x5. Extract All Employee Attendance Data
Attendance report data for all employees from the past 14 days is automatically extracted.
x6. Extract Applicant’s Attendance Data
From the data extracted in “x5. Extract All Employee Attendance Data,” only the applicant’s data is automatically extracted.
x7. Process Attendance Data 1
Unnecessary data is automatically deleted from the data extracted in “x6. Extract Applicant’s Attendance Data.”
x8. Process Attendance Data 2
The data processed in “x7. Process Attendance Data 1” is converted into Markdown format.
1. Report Clock-In Time
Employees enter their clock-in time when they begin work.
x1. “Working” Status
The “Attendance Status” data field is set to “Working.”
2. Report Clock-Out Time
Employees enter their break time and clock-out time when they finish work. Their work hours are automatically calculated.
x2. “Work Finished” Status
The “Attendance Status” data field is set to “Work Finished.”
x4. Clock-In/Out Data AI Evaluation
AI evaluates the clock-in/out data and outputs the evaluation results.
3. Confirm Work Hours
The employee’s supervisor reviews the employee’s reported clock-in time, break time, clock-out time, and work hours. They can also view the AI-generated evaluation results from “x4. Clock-In/Out Data AI Evaluation.”
2x. Handle Revisions
Employees review the reason for the revision and correct their clock-in time, break time, and clock-out time.
x3. “Leave” Status
The “Attendance Status” data field is set to “Leave.”
Compare Before/After
3. Benefits
Improved Payroll Accuracy
By automatically extracting 14 days of past attendance data, employees can now easily spot errors in their own time reports. This boosts the accuracy of attendance reporting, which in turn reduces payroll errors. As a result, you’ll see fewer incorrect salary payments and a lighter workload for the accounting department.
Protected Company Credibility and Brand Value
Incorrect salary payments are a direct hit to a company’s credibility. By reducing these errors, you can proactively prevent trust issues and safeguard your brand value.
Enhanced Employee Self-Management
Regularly reviewing their own attendance data helps employees develop a stronger sense of self-management. This reinforces attendance discipline, which is expected to lead to an overall improvement in attendance rates and productivity.
4. Other Business Applications
The mechanism of automatically extracting past work performance data and enabling it to be used to perform tasks can be applied to the following business operations:
Customer Support
Automatically pulling up past inquiry histories makes it easier to understand a customer’s previous issues and interests. This allows for more accurate and personalized support, potentially leading to higher customer satisfaction.
Information Security Management
Regularly checking and reporting on system account registration becomes even more efficient by referencing past report histories. Looking at this history helps you clearly identify differences from previous reports, enabling a quicker response if any improper registrations occur.
Expense Management
When making purchase requests for items within your department, you’ll be able to refer to past expense usage. This helps you understand the current budget consumption, making it easier to decide if a purchase is necessary or which items to choose.
AI summarizes growing daily reports, cutting delays and manager workload.
1. Issue: Delayed Daily Report Feedback
The company’s marketing department uses daily reports to share work progress and challenges. However, a significant increase in team members (from 4 to 7) has led to a surge in report volume. This has created a major bottleneck for managers, who are now too busy to thoroughly review all reports and provide same-day feedback.
Consequently, daily report submission has become a mere formality, diminishing its value and purpose for team members. An attempt was made to solve this by creating a standardized daily report template. Unfortunately, its focus on generality meant it couldn’t adequately capture the diverse tasks of each team member.
This forced employees to contort their work details to fit the template, ultimately increasing their burden and causing confusion.
To directly address the feedback delay and reporting burden, the process owner, who is also the manager, has successfully integrated AI-powered daily report summarization into their workflow.A key enhancement implemented was to have the AI output these summaries in Markdown format. This significantly improves readability, allowing the manager to quickly grasp the essential points of each team member’s daily report.
This strategic move enables the manager to rapidly check each team member’s report, transforming a time-consuming task into an efficient review process.
Daily report summaries enable managers to provide faster, same-day feedback.
Before
View details of the workflow diagram
Automated process starts at 7:00 AM.
1x. Business Day Check
If it’s a non-business day, the process branches and ends
1.Daily Report Input
2.Review
3.Resubmission (if returned for revisions)
After
View details of the workflow diagram
Automated process starts at 7:00 AM.
1x. Business Day Check
If it’s a non-business day, the process branches and ends
Auto-generated, locked titles prevent prefix errors and improve data reliability.
1. Issue: Forgetting to update the subject line
The company is a pharmaceutical organization specializing in the research and development of new drugs. Due to new scientific findings, technological advancements, and the need to ensure safety, internal regulations are frequently reviewed and updated to comply with the latest rules and standards.
Most revisions or abolitions of regulations are done by modifying and resubmitting previously approved regulations. Approved regulations are automatically prefixed* with “[Approved]” at the beginning of their subject line (e.g., “[Approved] Board of Directors Regulations”). However, there have been frequent instances of editing errors in the subject lines, specifically when this prefix was not removed when regulations were being reused and resubmitted.
* A method of making the type of matter easily identifiable at a glance by adding information indicating the category to the beginning of the subject line.
2. Solution: Automatic Subject Line Generation
To solve this issue, the process owner set the subject line editing permission to ‘Display Only’ (non-editable). Additionally, they designed the system so that the subject line is automatically populated when an application proceeds to the next stage.
Specifically, the system applies a format where the regulation name entered in the previous step is prefixed with “[Draft]” (e.g., “[Draft] Board of Directors Regulations”).
Reducing manual input errors enhances data accuracy and business reliability.
Before
View details of the workflow diagram1. Draft Regulation Input & Reviewer Nomination
The drafter inputs the draft regulation and nominates reviewers.
AI Check: Differences/Typos
If an old regulation is input, the AI automatically creates a comparison table between the new and old regulations.
2. Review Draft Regulation
When a draft regulation is submitted (including resubmissions), the nominated reviewers receive a notification email and review the draft.
1X. Rejection Handling
If there’s a rejection in step 2, the drafter addresses it.
Subject Line: [Withdrawn]
If the draft is withdrawn after step 1X, the subject line of the matter is prefixed with [Withdrawn].
3. CEO Approval / Board of Directors Resolution
Following step 2, the CEO decides whether to approve or reject the draft.
Subject Line: [Approved]
If approved in step 3, the subject line of the matter is prefixed with [Approved].
Subject Line: [Rejected]
If rejected in step 3, the subject line of the matter is prefixed with [Rejected].
After
View details of the workflow diagram1. Draft Regulation Input & Reviewer Nomination
The drafter inputs the draft regulation and nominates reviewers.
Subject Line: [Draft]
In step 1, the subject line is automatically set using the Regulation Name that was entered, prefixed with [Draft].
AI Check: Differences/Typos
If an old regulation is input, the AI automatically creates a comparison table between the new and old regulations.
2. Review Draft Regulation
When a draft regulation is submitted (including resubmissions), the nominated reviewers receive a notification email and review the draft.
1X. Rejection Handling
If there’s a rejection in step 2, the drafter addresses it.
Subject Line: [Withdrawn]
If the draft is withdrawn after step 1X, the subject line of the matter is prefixed with [Withdrawn].
3. CEO Approval / Board of Directors Resolution
Following step 2, the CEO decides whether to approve or reject the draft.
Subject Line: [Approved]
If approved in step 3, the subject line of the matter is prefixed with [Approved].
Subject Line: [Rejected]
If rejected in step 3, the subject line of the matter is prefixed with [Rejected].
At Dodo Cloud, shareholders cast their votes during general meetings either through a web form or by show of hands. Previously, a recurring issue arose where new responses that were submitted after the secretariat had already tallied the votes led to discrepancies that necessitated re-tabulation.
To address this, we’ve implemented a new operational rule and revised our workflow: web forms for resolutions are now immediately closed once voting on that agenda item concludes during the meeting. This measure prevents any further submissions after the deadline, ensuring the accuracy of vote counts and significantly reducing the workload for the secretariat.
1. Issue: Inaccurate Vote Counts
Dodo Cloud, an IT startup with around 20 shareholders, manages its general meeting voting as follows: The secretariat first emails the meeting agenda to shareholders in advance. Shareholders can then cast their votes on each resolution through a web form, accessible via email, either before or during the meeting. For those attending in person, voting by a show of hands is also an option.
During the general meeting, voting on each agenda item occurs the moment the chairperson states, “Those in favor of this resolution, please raise your hand.” At this point, the number of votes from shareholders who raised their hands (in-person) and the number of votes submitted via the web form must be tallied.
However, a recurring issue has been the submission of new web form responses after the secretariat has already compiled the for/against tallies. This leads to discrepancies in the vote count, making it impossible to finalize accurate voting results. Consequently, the secretariat is often forced to re-tally votes, significantly increasing their workload.
2. Solution: Closing the Web Form After the Deadline
The secretariat has established an operational rule to immediately close the web form for voting on any agenda item, once its deliberation and vote conclude during the general meeting.
The process owner will revise the workflow to ensure the secretariat blocks the web form at the appropriate moment, specifically right after the relevant resolution is voted on. Furthermore, the system will be configured so that the shareholder voting process automatically ends once the web form is closed.
This new measure will effectively prevent shareholders from submitting responses after the deadline, ensuring the integrity of the vote count.
Before :
View details of the workflow diagram
Process Initiation
Email Distribution (to shareholders)
Response Submission (via web form)
Status Update
Upon submission, status changes from “Unexercised” to “Exercised.”
After :
View details of the workflow diagram
Process Initiation
1. Proxy Voting Hard Deadline
Email Distribution (to shareholders)
Response Submission (via web form)
Status Update
Upon submission, status changes from “Unexercised” to “Exercised.”
*1. The process ends at one of the following points: the forced deadline for exercising voting rights, the entry of approval or disapproval, or the conclusion of the general meeting. (All parallel tokens will also be terminated.)
Compare Before/After
Reference:Shareholder Meeting Agenda Registration Process
*The Shareholder Meeting Voting Process is initiated from its parent process, the “Shareholder Meeting Proposal Registration Process”.
3. Benefits
Enhanced Accuracy of Vote Tabulation
By implementing measures to prevent the submission of responses after the designated tabulation deadline, the potential for vote fluctuations following the secretariat’s initial count has been eliminated.
This enhancement directly contributes to the improved reliability of voting outcomes.
Reduced Operational Burden on the Secretariat
The elimination of the need for re-tabulation has significantly alleviated the workload on the secretariat.
This operational improvement has led to increased efficiency in the administration of general meetings.
Zero Unnecessary Steps! Start Editing Without Requests, for Smooth Updates
The Marketing Department at DonDon Web Co.Ltd was experiencing inefficiencies in its web content update process. Previously, even when an editor identified a need for an update, they first had to register the task as a “request”. This unnecessary step led to delays and duplication of work.
To resolve this, the process owner introduced a new, streamlined workflow. Now, editors can directly initiate work without having to register a request. This change has significantly boosted operational efficiency by allowing editors to immediately begin editing.
* This process improvement story is fictional and not related to any real person or organization.
1. Issue: Editors inefficiently submit their own work requests.
DonDon Web Co.Ltd is a web development company with a team of 50. Their Marketing Department handles daily updates for web pages, ad creatives, and promotional materials.
Typically, when the Marketing Department gets a web update request, an available team member takes on the task and edits the content. After completing the work, another editor reviews it.
However, a problem arose when an editor noticed a page needed updating themselves. They still had to register the task as a request from a “requester” first. This extra step wasn’t just a burden; sometimes, another team member would pick up the same task, preventing the original editor from being able to edit the task.
2. Solution: Implement a Workflow Allowing Editors to Initiate Tasks Directly
The process owner introduced a new editing workflow to allow editors to make their own judgments and start work immediately. This revision to the traditional “request registration → editing” flow means editors can now begin editing directly, without having to go through the request registration process.
Before
View details of the workflow diagram0. Request
The requester initiates the task and submits an editing request.
1. Edit
The editor begins the task and performs the requested edits.
2. Review
The reviewer checks the content created by the editor for any issues. If corrections are needed, they request the editor to re-edit.
1x. Re-edit
The editor performs further edits based on feedback from the reviewer.
After
View details of the workflow diagram0. Request
The requester initiates the task and submits an editing request.
1. Edit
The editor begins the task and performs the requested edits.
1a. Edit (Self-Initiated)
The editor proactively starts the editing work.
1b. Edit (Request-Driven)
The editor performs the editing work in response to a request from a requester.
2. Review
The reviewer checks the content created by the editor for any issues. If corrections are needed, they request the editor to re-edit.
1x. Re-edit
The editor performs further edits based on feedback from the reviewer.
3. Benefits
We’ve eliminated the wasteful step of self-requesting tasks
Eliminated the hassle of registering requests, allowing for faster operations.
Prevents duplicate editing tasks and reduces operational waste.
Ensures the entire process runs smoothly, preventing operational stagnation.
4. Other Business Applications
Design Revisions
Designers can start making corrections immediately, without waiting for instructions from clients or directors.
Internal Document Updates
Necessary revisions are reflected quickly, helping to keep operational manuals current.
Product Information Changes
Updates to sales pages and catalogs can be made smoothly.
Bug Fixes
You can establish a system where developers can immediately correct bugs they discover.
