Home > Product Work > Falak AI Assistant
Falak: an AI assistant that shows its sources
Concept, Research & Coded Prototype
A research-led study of how an AI assistant in a bank should handle uncertainty, hand over to people, and keep human oversight honest. Eight interaction patterns, each checked against the 2019–2026 literature, built as a Flutter prototype for left-to-right and right-to-left layouts (English and Arabic). Falak is a fictional bank; every name and figure is invented.
Four ways an AI answer earns trust
Falak is a coded prototype of a banking assistant that shows where each answer comes from, how sure it is, and hands over to a person when it should. It runs four scripted examples: a balance (a fact), a spending estimate, a loan question an AI shouldn't decide, and a staff view where a relationship manager signs off an AI suggestion.
One finding shaped everything. Trust signals such as percentages, citations, visible steps and explanations raise trust even when the AI is wrong. They only make trust appropriate when they lower the cost of checking an answer, or push the person to form their own judgement first.
The Challenge
My Role
Concept, research and prototype
- Defined the eight interaction patterns of the first PoC.
- Validated each against the literature; three were reworked, one dropped.
- Designed the interaction, visual language and bilingual copy.
- Built the prototype in Flutter with Claude Code.
Scope
- Two audiences: customers and bank staff
- Left-to-right and right-to-left layouts (English and Arabic)
- Four scripted scenarios, light and dark
- No backend, no live model: the conversations are scripted to study the patterns
Approach
Eight patterns. Each one either changes what the system does, or makes checking cheaper.
Three behaviours, not one chat
Every question lands in one of three tiers. A fact read from the system is answered straight away. An estimate is softened and broken down. A question an AI shouldn't decide goes to a person, with no best guess attached. The tier is set by the kind of question and the data available, not by a confidence number the model reports about itself.
Words and marks instead of a percentage
No "87% confident". Each answer carries a mark borrowed from banknote printing: solid for data read from the system, hatched for an estimate, dotted for something not yet synced. Estimates are shown as a range, and the estimated and unsynced amounts open the transactions behind them in one tap.
A source on every answer, the reasoning one tap away
One line lists the sources actually used and how fresh each is. "Why this answer" is collapsed by default and lists the data the answer was built on, not a narrative of thinking.
No fake progress
A balance arrives immediately. Steps appear only when the assistant consults several sources, each step names what it is reading, and the work can be stopped. No artificial delays to make the answer look considered.
A person is always one tap away
"Talk to a person" sits in the header. The loan question is handed to a licensed advisor with an honest wait time, a booking alternative, and the conversation passed along so nothing has to be repeated. Any answer can be flagged as wrong.
Human oversight that isn't a rubber stamp
The relationship manager sees the client data and answers "should we contact them?" before the AI suggestion appears. Three named checks replace a generic "I have reviewed" box. Approving needs a reason exactly as declining does. The form then turns into a receipt of what was logged. Inferred data ("recently married") is labelled as inferred, sensitive and not confirmed.
Same signals in English and Arabic
Switching language translates the whole conversation, screen-reader labels included, and flips the layout to right-to-left. Numbers stay unmirrored. The marks, sources and tiers are identical in both languages: culture changes how much people trust at the start, not the mechanism by which trust follows reliability.
Red means error or decline, nothing else
In banking red already means overdrafts, refusals and fraud. Uncertainty is never shown in red, and colour is never the only cue. Text contrast is at least 4.5:1 in light and dark, keyboard focus is visible, and motion is reduced when the system asks for it.
Key Decisions & Trade-offs
The first PoC had all eight patterns. The literature confirmed four, reworked three and replaced one. Every row links to the evidence.
| Decision Point | Options Considered | Why We Chose | Evidence |
|---|---|---|---|
| How sure the answer is | 1. Always-visible numeric score with a traffic light 2. Verbal bands tied to an action, number on request 3. Marks (solid / hatched / dotted) and ranges, no number | With displayed confidence inflated and accuracy held fixed, people followed the AI more and most rated the overconfident AI as well calibrated. LLM self-reported confidence is systematically overestimated. Marks read without colour and without a reference class to misread. | Li et al. 2024 · PAIR Guidebook |
| Estimates | 1. A precise total next to uncertain parts 2. A range with the parts broken down | A numeric range barely lowers trust in the source, while a single precise figure placed next to uncertain components distorts judgement. The breakdown tells the user where the uncertainty sits and what to check. | van der Bles et al. 2020 · Kale, Kay & Hullman |
| Cost of checking | 1. More trust signals on the answer 2. One tap to the transactions behind an estimate | What reduces over-reliance on AI is making verification cheaper, not adding more signals. Every estimated or unsynced amount opens its items. | Vasconcelos et al. 2023 |
| Sources | 1. A fixed list of three sources on every answer 2. Only the sources actually used, with freshness | Citations raise trust even when they are random, and under 10% of people check them. A source label is only worth showing if it is exact. Sources did reduce reliance on wrong answers, where explanations increased it. | Ding et al. 2025 · Kim et al. 2025 |
| Visible steps | 1. Scripted 'streaming validation' with fixed delays 2. Steps only for real multi-source work, stoppable | Slow algorithmic answers are judged less accurate, unlike slow human ones. A fake 'Authenticating credentials…' step is a false claim about a security check. AWS guidance: a step is a unit of real work; stream fast answers directly. | Efendić et al. · Cloudscape GenAI patterns |
| Out-of-scope questions | 1. Best guess with a low confidence label 2. Handoff to a person, no guess shown | Telling people the AI had declined, without showing its uncertain guess, gave the most accurate decisions. Showing the guess pulled people towards it even when it was wrong. | Bondi et al. 2022 |
| Staff checkpoint | 1. Generic 'I have reviewed' attestation, reason only on decline 2. Own read first, named checks, reason on both paths, receipt | Automation bias hits experts and novices alike and isn't fixed by training. An independent judgement before seeing the suggestion reduces over-reliance more than explanations do. Asking for a reason only on decline makes approving the path of least resistance. | Parasuraman & Manzey · Buçinca et al. · Green |
| Cultural framing | 1. Different trust mechanisms per language (high/low context) 2. Same signals in both languages, style as a user preference | Hall's high/low-context model isn't validated as a country classification, and Gulf preferences differ between countries. Across cultures trust starts at different levels but follows reliability the same way. The real risk is technical: calibration error is far worse in Arabic than in English. | Chien et al. · Zhou et al. 2025 |
How sure the answer is
Estimates
Cost of checking
Sources
Visible steps
Out-of-scope questions
Staff checkpoint
Cultural framing
The prototype on mobile
What was delivered
What I Learned
Trust is the wrong success metric
Almost every signal in the literature (percentages, citations, visible steps, reasoning) raises trust even on wrong answers. The patterns that hold up either move the decision from the user's interpretation to the system's behaviour, or make checking cheaper. A PoC should be judged on behaviour, not on a trust questionnaire.
A scripted prototype can't measure calibration
Falak always behaves the same, so it tells you nothing about how well people calibrate their trust. The next step is a test with deliberately wrong answers: how many errors people catch, how often they open the sources, how long staff reviews take and how much they vary.
The evidence has gaps the bank could fill
Most studies are lab or crowdsourced experiments on simplified tasks. There is no field study of banking chat, no attribution test on internal data, and no calibration measure for Gulf Arabic. A bank running this with real data would be producing new evidence, not confirming old.
Regulation sets a floor, not a design
The AI disclosure in the first message is the one duty that applies today (EU AI Act art. 50). Human oversight for high-risk credit decisions is required later, but GDPR art. 22 and the SCHUFA ruling already apply. The law says oversight must exist; it doesn't say how to keep it from becoming a stamp. That is the design problem.
Regulatory Reference
The rules the design was checked against, as they stand after the EU Digital Omnibus. A design map, not legal advice. No rule found requires citing the data source on every answer: that pattern is good practice, not a duty.
EU AI Act art. 50(1)
EU AI Act art. 50(2)
Digital Omnibus on AI
EU AI Act art. 14(4)
EU AI Act art. 86
EU AI Act art. 5
GDPR art. 22 / SCHUFA ruling
QCB AI Guideline (Qatar, Sep 2024)
CFPB Issue Spotlight (US, Jun 2023)
Sources: Freshfields on the Digital Omnibus · Usercentrics · BSI on art. 50 · art. 14 · art. 86 · A&O Shearman on SCHUFA · QCB guideline · CFPB · Bird & Bird on the EBA mapping. The EBA mapping (Nov 2025) finds no significant conflict between the AI Act and banking law, and lists as high-risk only creditworthiness assessment and credit scoring of natural persons.
Try it, or talk about it
The demo runs in the browser, in English and Arabic. If you are building an assistant where being wrong is expensive, I'd like to compare notes.