Designing trust into an AI that builds software — AI Next Solution Studio case study

Business impact at a glance

Users stopped second-guessing the AI and started approving it

ProblemEnterprise users wouldn't approve AI-generated solutions they couldn't verify.

ChangeEvery AI claim became something a person could check, correct, or challenge.

ResultUsers finished the task more often, rated the product as more usable, and approved with less hesitation.

Task completion

62.5%→0%

Pre-GA prototype · moderated · same task
Round 1 n=8 → round 2 n=12 (8 returning + 4 new), 3 weeks apart

System Usability Scale

58.4→0

Same pre-GA sessions · crossed the 68 industry average

Progression to Test

0%

Production · early-access clients
312 solutions, first 6 weeks of GA

Governance users more willing to approve without opening every assumption

0of 6

Pre-GA · 6 participants in governance-heavy roles

Went to the structured record, not chat, to find a past decision

0of 12

Pre-GA retest · all 12 round-2 participants

Qualitative outcomes

Less ambiguity in approval decisions (PM-reported)
More confidence in AI-generated solutions (round-2 sentiment)
Teams challenge and refine proposals before build

Method note: usability results come from small moderated studies (8 participants, then 12, three weeks apart) and should be read as directional. GA data covers one early-access cohort with no pre-launch baseline, so progression to Test is a starting benchmark, not a before/after comparison.

The moment that changed

Same AI. Same output. A completely different decision.

The whole project comes down to the few seconds before someone clicks Approve. Before, they were asked to sign off on a paragraph. After, they could check every claim the AI made.

Before · round 1Task completion 62.5%

AI recommendation→Approve

AIDiscovery summaryDraft

● Sent back for manual review

“I don't know why the AI recommended this…”

After · shippedTask completion 91.7%

Discovery Snapshot→Assumptions→Business outcome→Approve

AIDiscovery SnapshotReady for review

Use case ✓

Club Management System

Business outcome ✓

Eliminate paper-based record-keeping

Systems ✓

ClubAdminApp

Data ✓

MembershipRequest, Event

Teams ✓

SocietyMember, SocietyManager

!Assumptions the AI made, logged for audit0

✓ Approved · moving to Reimagine

“

I don't know why the AI recommended this, so I'm hesitant to approve it.

PARTICIPANT · ROUND 1 USABILITY TEST

→

“

I can see the assumptions and reasoning, which makes me comfortable moving forward.

PARTICIPANT · ROUND 2 RETEST

Research overview

Grounded in the people who'd have to sign off

Research at a glance4 studies

0

Contextual interviews

0

Users, initial usability test

0

Users, retest (8 returning + 4 new)

0

Real solutions analysed post-GA

WhoBusiness analysts, process owners and citizen developers across banking and retail pilot accounts

Tasks

Take a real process from your own job to an approved brief
Find a decision made earlier in the session
Decide whether to approve using only the assumptions count (6 governance-role users)

“I don't trust a system that just tells me ‘done.’ I need to know what it assumed, because I'm the one who gets asked questions six months later.”

P7 · PROCESS OWNER, RETAIL OPS

“Every low-code tool either treats me like I can't understand the technical side, or dumps raw JSON on me. There's no middle ground.”

P11 · CITIZEN DEVELOPER

“By the time the requirements doc is reviewed and handed to dev, the business has changed its mind about half of it.”

P3 · BUSINESS ANALYST, BANKING

Study ledgerWhere every number comes from

Study	Stage	Sample	Who	Used for
Contextual interviews	Pre-design	14	Business analysts, process owners, citizen developers · banking & retail pilot accounts	Problem framing, P3 / P7 / P11 quotes
Usability test, round 1	Pre-GA prototype	8	Users each bringing a real process from their own job	62.5% completion, SUS 58.4
Assumptions-register test	Pre-GA prototype	6	Participants in governance-heavy roles	5 of 6 more willing to approve
Usability retest, round 2	Pre-GA prototype	12	8 returning + 4 new participants, same task	91.7% completion, SUS 76.2, 12 of 12 findability
Phase-completion funnel	Production	312 solutions	Every solution started by early-access clients, first 6 weeks of GA	48.4% reached Test, Build drop-off, 3.4× dwell

Evidence chain

Three design decisions, each traced from research to result

Each decision below is traced from the research that prompted it, through the insight and the design change, to how it was measured.

01Discovery Snapshot — five falsifiable fields

ResearchFirst test (n=8): 62.5% completion, SUS 58.4. Three participants couldn't tell what the AI understood from what it echoed back.

InsightThe issue wasn't accuracy. Users couldn't see what the AI had understood, so they couldn't check it.

DecisionSplit the AI summary into five independently editable fields: use case, outcome, systems, data, teams.

OutcomeRetest (n=12): completion 91.7%, SUS 76.2. Sentiment moved to “I can just fix this field.”

Shipped Discovery Snapshot with five editable cards

Shipped · five separately editable cards

Task completion62.5% → 91.7%

Round 1

Round 2

SUS58.4 → 76.2

Round 1

avg 68

Round 2

02Assumptions register

ResearchP7, a retail process owner, is the person who gets questioned “six months later” when something breaks.

InsightApproving carries personal accountability, so the AI's own judgement calls need a record.

DecisionLog every assumption the AI infers and surface a running count in the conversation.

Outcome5 of 6 governance users were more willing to approve. Open gap: the count had no drill-down.

03Chat kept separate from the record

ResearchInterviews (n=14): decisions got lost between conversation and documentation, and requirements drifted before handoff (P3).

InsightWhat was discussed and what was decided are different things, and users need to see which is which.

DecisionEvery phase pairs a temporary chat with a persistent, structured record of what was decided.

Outcome12 of 12 went to the structured panel, not the chat history, to find a decision made moments earlier.

In the wild

What 312 real solutions told us

Six weeks into GA we pulled a phase-completion funnel across every solution started by early-access clients. The redesigned front half held up. The back half exposed the next problem.

Discover

100%

Reimagine

89.4%−10.6% · mostly one-off explorations, not friction

Design

78.2%

Build

63.5%−14.7% · largest drop in the funnel

Test

48.4%

3.4×longer dwell time in Build

Build is where the interface silently switched from conversation to dashboard — with nothing to prepare users for it.

Beyond launch

What we're tracking next

Six weeks of GA is a starting point, not a verdict. These are the longer-term business outcomes the team is measuring against the launch numbers.

Review cycle time

Prompt → approved brief

BaselineNot yet reported

Abandonment before Design

21.8%

BaselineWeeks 1–6 of GA

Progression to Test

48.4%

BaselineWeeks 1–6 of GA

Approval confidence

5 of 6

BaselinePre-GA; to re-measure in production

Tradeoffs

What didn't work, and what I'm doing about it

Explored · cut

Showing more reasoning made trust worse

We first exposed more of the AI's reasoning. Testing showed information overload: users preferred short explanations with clearly stated assumptions.

Explored

Overload

Shipped

Concise + assumptions

Learned: what builds trust is how much people can check, not how much you show.

Shipped · incomplete

A count isn't an audit

The register shipped as a number with no drill-down. P2: “Numbers I can't check aren't audits, they're just… vibes.”

Assumptions 24

Managers approve within 48hConfirmChallenge

One shared membership databaseConfirmChallenge

Next: expand the count into individual assumptions you can confirm or challenge in place.

Would reverse

Shipping Build with no bridge

We treated the move from Design to Build as visual polish. It was really a change in how the product works, and the funnel's biggest drop sits right there.

→Moving from conversation to build

Components build in parallel, not in chat
Pause or leave anytime — we'll notify you

Next: one framing moment before Build, then a build view you can watch, pause, and resume.

The shipped product

Six real screens, one thread of trust

Live product, not mockups. Each pin marks the detail the research above is about.

studio / home

01 / 06

Designing trust into an AI that builds your software