HOW WELL IT WORKS
Every number below carries the command that produced it
Measured against the live APIs
- 1368 ms2026-09-06
Time to interrupt, end of speech to first sound
p50 over ten interventions, re-measured after the anchor fix. Min 1278 ms, max 1538 ms, and all ten carried a full end-to-end total. Budget is 2500 ms.
Ten live interventions driven by a probe recording, measured from the end of the advisor's speech in the recogniser's own audio timeline to Saakshi's first audio frame.
pnpm test:e2e:latency
- about 1.2 s2026-09-06
Of that, recogniser endpointing
The wait before the turn is finalized dominates. Saakshi's own reply is roughly 210 to 320 ms of the total.
Breakdown logged per intervention alongside the total.
pnpm test:e2e:latency
- 2226 ms2026-09-06
Judge-solo, one person drives the whole demo
Roles bound, six advisor turns all from the recording, four of eight disclosures ticked, the planted claim interrupted.
A pre-rendered advisor mixed into the microphone stream, with only the customer played from a fixture.
pnpm test:e2e:judge-solo
- 1.00 / 1.002026-09-11
LLM layer, prohibited claims and disclosures
Ten labelled dialogues, all ten reached a model (Groq openai/gpt-oss-120b with a strict JSON schema; one fell back to gpt-oss-20b). Violations: precision 1.00, recall 1.00. Disclosures: recall 1.00, up from 0.00 on the gateway's only model. Latency p50 1268 ms. The analyzer is advisory since 2026-09-09: its findings are notes for a reviewer and never tick a card, flag a claim or get spoken.
Ten labelled dialogues through the configured LLM endpoints (Groq first, AssemblyAI LLM Gateway last) with a strict JSON schema.
pnpm eval:analyzer
- 1.7 s2026-09-11
Teach-back question generation
One call per session on Groq openai/gpt-oss-20b with a strict JSON schema (gpt-oss-120b is tried first). Five of five questions passed validation, all five from the model, each with a Hindi hint. Was 5.4 s with prose-described JSON on the gateway's only model.
One live call, then the same validation the room applies before speaking a question.
pnpm eval:questions
- 0 of 122026-09-09
Disclosures ticked that were never made
A recording where the advisor makes four of eight disclosures and says acoustic neighbours of the other four (free book, fund manager, tees at the golf club, surrounding valley, illustrate the layout, location of our office). Across three recogniser vocabulary modes, the four omitted cards stayed empty every time; the four true ones ticked every time. Identity-only vocabulary, now the default, kept 8 of 8 on the demo script.
Same WAV through the fake microphone under identity, full and none keyterms; board read from the room state; reports in eval/keyterms/.
SAAKSHI_KEYTERMS_MODE=identity SAAKSHI_FAKE_WAV=tests/fixtures/omitted-draft.wav pnpm test:e2e:keyterms
- $0.162026-09-09
Cost of one demo session
Four sessions of about 2 min 15 s: Streaming STT billed 112 to 113 s and the Voice Agent 103 to 113 s, at list prices ($0.72 an hour for STT with speaker labels, prompting and voice focus; $4.50 an hour for the Voice Agent). The Voice Agent is about 86 percent of it. LLM Gateway calls add under a tenth of a cent. Not yet reconciled with the dashboard.
Billed seconds reported by both APIs at close (Termination and session.ended), multiplied by list prices.
pnpm test:e2e:keyterms
Measurements taken against the live AssemblyAI APIs. Each carries the command that produced it, so anyone can reproduce or contradict it. Updated by hand when re-measured; the method is in docs/decisions.md.
The rule engine against a labelled corpus
| What | Precision | Recall | F1 | Found | Wrongly flagged | Missed |
|---|---|---|---|---|---|---|
| Prohibited claims | 100.0% | 100.0% | 100.0% | 31 | 0 | 0 |
| Required disclosures | 100.0% | 100.0% | 100.0% | 29 | 0 | 0 |
- Labelled turns
- 70
- insurance-ulip-in
- 35
- loan-kfs-in
- 35
- Disagreements
- 0
Read this as a regression suite, not as an unbiased estimate of how the engine behaves on unseen speech. The corpus was labelled first, by hand, from each sentence alone. The engine then disagreed on eighteen of them, nine patterns were genuinely wrong and were fixed, and one label was too strict and was corrected. A score of 100.0% means the patterns now agree with those labels, and that a future edit which breaks one of them will be caught. It does not mean the next unseen sentence will be handled correctly.
pnpm eval · report written to eval/report.json
What is not measured
- Teach-back answer recording has never run against a live human. The tools and the guards are unit tested, and the golden recording cannot exercise them.
- Diarization accuracy is measured on synthetic voices only. A real two-person session is still owed.
- Firefox and Safari are untested. Chromium is the supported browser today.
- Cost per session is computed from the billed seconds both APIs report, at list prices. It has not yet been reconciled with the dashboard.
- The analyzer is advisory since 2026-09-09; its notes are shown but their usefulness to a reviewer has not been measured.