J.A.R.V.I.S.

The problem

Every AI chat is a text box. Nothing from the machine reaches it, and a long conversation is one scroll where the third subject buries the first. And the moment a product holds people's API keys it needs a server, accounts and billing, and their questions pass through someone else's logs.

What I owned

• Product, design and implementation end to end — the console, its server and the serverless web version
• The board: every subject is a window, threads dropped together or grouped on request share a bubble, and a context web — worked out locally, with no model call — shows what else on the board shares a thread's subject
• One back end shared by both versions, handed only a place to read keys from, so the PC and the web page cannot drift apart
• The voice: the opening sentence sent for synthesis the moment it is complete, the rest gathered into one passage sent just before the voice would run dry, all scheduled back to back on the audio clock, so he starts talking about a second after the first words
• A command parser so everything you can click can also be said — acting only when a command is the whole of what was said — and a fixed set of directives the model may use to operate the console itself
• Web search only when a question needs it, and a Quick / Balanced / Thorough thinking setting per service: a greeting went from 7.8 to 2.8 seconds
• Real telemetry on the PC — per-core CPU, the GPU, a network sweep — and the honest equivalents a browser can measure for the web version
• The first-run guide, the QA log (127 numbered fixes), 118 unit tests and 21 end-to-end scenarios run in CI against a fake service, and the docs

Technical challenges

• Speaking while the answer is still streaming: where a sentence ends, what to say first when the opening runs long, and holding the last quarter-second back to trim the silence a service leaves at the join
• Free tiers run out under real use — web grounding first, then the voice — so an account's voice and hearing models are tried in turn and a spent one rests; a spent search allowance means an answer without the web, a spent voice the device's own; every refusal is said in plain words, and the other service answers if nothing has been said yet
• A version with no server at all: the browser measures what it genuinely can, and a Content Security Policy lets the page's requests reach only the two services and a few keyless reading sources
• Where an answer belongs: a local guess, then the model's own choice, each sent replies to the wrong place — so every answer now goes into the thread in front, the console decides, and the model only names a new thread, with a directive

Engineering decisions

• Bring your own key, kept on the device: no accounts, no billing, no server costs, and nothing to leak
• Two services only, Gemini first because its free tier needs no card; speech models running in the browser were built, measured and dropped — five to ten times slower than the services' own voice and hearing
• The connection layer takes its services as a parameter, so it is unit-tested with fakes rather than with a key
• Vanilla TypeScript, no UI framework: windows are plain DOM over a canvas that draws the core, and a reading is written only when it changes
• Free and open source under AGPL-3.0, supported by a coffee rather than a subscription

Technology