outcome and takeaway
81%
The outcome led to an 81% improvement in productivity, resulting in a decrease from 60 min to 5 min per task.
NPS score
The final design outcome helped client gain an improved NPS score among its target users.
Clearly outlining the capabilities and limitations of the product to set realistic user expectations and avoid unintended deception
Discuss with XFN stakeholders all the tradeoffs of collecting or not collecting different types of feedback.
up to date news
85%
A Fortune 10 company automates contract and invoice processing with Orby AI, resulting in an 85% improvement in productivity
60%
SV large tech company uses Orby AI to accelerate migration to a new contract management tool, reducing costs by 60% and migration time by 70%
After implementing Orby AI, the customer was able to increase cost savings and productivity by 85% and speed up processing time by over 70% while simultaneously improving supplier and employee experience. A 4X ROI was achieved for the project within days.
design process
Make the validation process streamlined
Based on the competitive analysis, we discovered that most document processing tools share a common layout that users are comfortable with. While there is no need to invent a new pattern, a few changes to the navigation could significantly enhance the speed of finding and editing incorrect suggestions.
Improve Key term list navigation
The primary goal of validation is to identify errors from AI-generated results. Leveraging the model's ability to detect problematic outcomes, we implemented filtering tabs for users to address issues first and complete the rest efficiently.
v1
v2
Final
From Internal Testing
Show confidence score by risk level and show score for each term are not helpful to identify problems
Validate a little bit faster
In many document processing products, actions are integrated within a key term list on the left panel, leading to a complex visual queue between error detection and editing. To simplify this process, we made actions more contextual and quicker through a floating modal that appears alongside the content.
Research Finding
A complex visual queue between document and key term list makes it challenging to navigate
Before
User need to constantly compare list content with document
After
Floating modal allows user to focus on the document side
Make it effortless to start with
Guide user to start training
Set expectations for training
Interactive step-by-step guide
Provide visibility into the progress
Training process pose challenges
Before Orby can achieve improved automation accuracy, the model require further training from users, involving the completion of 100 - 200 manual extraction tasks. Also, training for AI is a new concept for many users. Guiding users through this process smoothly while maintaining their patience is crucial.
Strategies for navigable training
Show users progress and give feedbacks throughout the training
Break down the training into interactive steps with a sample task
Access to Orby's support through a dialogue feature
Make it responsible to build trust
How to make confidence score easy to understand?
Orby can provide a confidence score for each suggested task and term, indicating the model's level of confidence in its accuracy. Our design goal is to assist users in prioritizing issues, so after iterating through various solutions, we've opted to only highlight tasks and terms with low confidence scores.
Final
v1
v2
What is the difference between 82 and 89?
show all levels are unnecessary
How to balance user edit & automation to calibrate trust?
The predictions aren't always 100% accurate. To rectify any incorrect or missing predictions, we've introduced a floating modal for flexible edits, ensuring precise results through collaboration between the product and the user.
Flexible fix for wrong suggestions
Add missing contents
show what might be inaccurate
show what might be missing
Let the user take back control when needed
If the suggested term or the entire task is totally wrong or unwanted, users can easily locate the "Decline a term" or "Discard a task" action for quick resolution.
if the term needs refinement
if the term does not exists
Let the user feel confirmed before proceeding
Upon completion of the current workflow, we introduce an additional step for users to review the final output—the spreadsheet—to ensure task completion. This small step helps to reinforce user trust in AI co-working mode.
next workflow
Include opportunities for users to provide feedback
When the system behaves in a way that a user doesn’t expect or want, the user should have an option to share feedback. And the feedback can be used to improve the AI model. When presented with an error message as well as alongside “correct” system output, it is good time for users to “complain” and give feedbacks.
A follow up review directly after a queue of documents automation can remind users of AI’ credits and collect feedback.
Acknowledging users how their feedback will influence their future experience can help with more efficient data collecting.
Research
Started with limited knowledge in use case and AI automation, we navigated the problem space through research:
To understand how to connect business goal and technical feasibility with design, we researched on AI Automation’s strengths and limitations.
We conducted an analysis of relevant products, including RPA, contract processing, and document assistants, to learn from existing patterns, user behaviors and identify areas for optimization.
We got stakeholders’ perspectives over the product expectation and narrowed down the design scope on key problems.
Key insights
AI cannot achieve perfect accuracy, and it's essential to ensure its responsible use by users.
Clear navigation pattern assists users in prioritizing key operations and enhances efficiency.
Stakeholders aim to ensure that new users can easily grasp and maximize the product's capabilities.
features
Guide user through training process
Provide visibility into the process and make the training process less effort to complete
Lack of intuitive flow to help expedite the processing Lack of intuitive flow to help expedite the processing
Automated document processing without the need for scripting
A clear navigation pattern for validation and identify problems
Validate with better accuracy and efficiency
AI automation creates a new landscape for enterprise document workflow
Orby AI is a web-based AI automation tool to enhance document handling productivity for enterprise clients. In this version, our client's focus is on the contract handling use case for financial teams. As a member of the design team, my primary responsibility is to translate AI's capabilities into a tangible and enhanced user experience.
It can empower document workflow by reducing maintenance costs compared to traditional methods, handling a wider range of tasks beyond rule-based processes, and resulting in more efficient document processing.
Input
Learn from user actions
Automated document processing
Improve from user validation
Automated Processing
Output
End User:
Financial Analysts
“I support the finance team to review documents and file reports efficiently and accurately.”
Learn users pattern
Automate documents faster
How can we establish trust in AI automations from the outset?
AI automations aren't always perfect. With more user input and feedback, accuracy improves over time. However, early failures might lead to user churn.
How to design an intuitive workflow to speed up processing?
After AI completes its tasks, users have to spend time verifying its accuracy. A quick and efficient experience is essential for the success of the product.
From AI's technical feasibility to a user-friendly product, there remain unresolved challenges:
document automation AI Agent
Orby.ai
My Role
Product Designer
customer (nda)
Web App/ AI
Enterprise Automation
customer (nda)
A Fortune 10 company
SV Large tech company
Tools
Figma
Slack
Notion
Timeline
2023
DesCription
AI agent that automates complex enterprise workflows, transforming how businesses handle repetitive tasks and boosting productivity benefits non-tech background users.
Context
In early 2023, I worked with Orby to create a groundbreaking AI tool for enterprise employees with no AI or development background. This was before ChatGPT's popularity, when Generative AI was largely unknown. We faced the challenge of developing a user-friendly product in uncharted territory, with no benchmarks and little public trust or awareness. Our goal was to demystify AI and make it accessible in the corporate world. I contributed to the core function - Document Automation AI Agent and the system dashboard design(also enclosed in my portfolio).
Due to NDA restrictions, I can only share limited details here. However, I'm eager to discuss the development process and real-world applications of Orby’s work. Contact me to learn more about this transformative project.