AI-Powered Decision Support
Building the first human-in-the-loop AI feature for a risk management platform and defining reusable interaction patterns for future AI systems.
Overview
I designed an AI-assisted evaluation tool into Capital One's risk management platform to improve data quality at the point of entry. The work delivered a production decision-support feature and defined reusable AI interaction patterns to support consistency across future features.
Key Decisions
Analysis Placement
Analysts needed to write control descriptions that satisfied compliance requirements. We embedded AI evaluation directly into the authoring experience so they could review findings without leaving their workflow, while keeping review and decision-making with the analyst.
Control Description
Complete the form below to describe the control.
Analysis Breakdown
The AI Analysis panel brought the overall evaluation into one place, including the AI context, individual findings, and the information used to support the analysis.
AI Analysis
AI Context
Positioned at the top of the hierarchy to establish immediate system intent. Incorporates scoping metadata and disclaimers to prevent users from treating AI outputs as absolute ground truth.
Report
Houses the core evaluative data. Surfaces multi-criteria binary checks (Detected / Not Detected) alongside inline inspect triggers (?) that allow analysts to audit granular rule matches and tooltips on demand without cognitive overload.
Attribution
Provides governance and source traceability. Anchors the AI's output back to specific institutional compliance frameworks to maintain audit readiness and human accountability.
Explaining individual findings
Each finding could be opened for a deeper explanation. Rather than showing only the result, the detail view exposed the confidence, conclusion, supporting evidence, reasoning, and underlying logic so analysts could evaluate the finding before deciding what to do next.
AI Confidence Indicator
Presenting AI confidence as a progress bar and percentage blurred the distinction between confidence and analytical results already shown throughout the interface. High, Medium, and Low communicated the same information while keeping confidence as supporting context.
AI Analysis
Quantitative Label
This approach was rejected because it displayed exact numerical probabilities (e.g., 87%) alongside complex chart elements, leading users to mistake model confidence for hard data metrics.
AI Analysis
Qualitative Label
This approach was selected because it communicates system certainty accurately using qualitative tiers (e.g., Low, Medium, and High) and remains distinct from empirical data metrics.
Status Indicators
Control descriptions were evaluated against multiple quality criteria. The AI generated a report showing whether each criterion was detected within the description.
Indicator Colors
Multiple color options were evaluated using established UI semantics to reinforce AI evaluation without implying success or failure.
| Option | Consideration | Result |
|---|---|---|
| Green / Red | Implied success and failure. | Rejected |
| Blue / Orange | Communicated information and warnings. | ✓ Final |
Indicator Terminology
Multiple terminology options were evaluated to communicate AI evaluation without implying certainty or correctness.
| Option | Consideration | Result |
|---|---|---|
| Pass / Fail | Implied a final judgment. | Rejected |
| Present / Missing | Described the criterion rather than the evaluation. | Rejected |
| Found / Not Found | Resembled search results. | Rejected |
| Detected / Not Detected | Framed the result as an evaluation rather than a definitive system status. | ✓ Final |
Impact
- Analysts incorporated the reasoning modal into their review workflow before making final decisions.
- Stakeholders reported improved clarity and faster review of AI-generated evaluations, with informal feedback indicating meaningful efficiency gains.
- Engineering handoff was completed on schedule, including full specifications for the reasoning modal and associated UI components.
Reflections
- Formal post-launch measurement was not in place, so long-term impact was not quantified at scale.
- Future work would include lightweight instrumentation earlier in the build process to capture usage and decision signals prior to launch.
- The primary constraint was not feature capability, but how AI outputs were structured and interpreted within the workflow.