Grace

The idea

Most voice assistants are a cloud service with a wake word. I wanted to see how far you can get with everything running locally on a gaming laptop. Grace has no wake word, you can interrupt her mid-sentence, and she keeps a long-term memory, so she can remember what I was working on yesterday.

The models

Grace runs on a gaming laptop with an RTX 5090 (24 GB) and an Intel Arc iGPU. The trick is to split the work so the models do not fight over the same graphics card.

• Hearing: Whisper large-v3 through OpenVINO, on the Intel iGPU. That leaves the whole NVIDIA GPU for the language model. Whisper reliably mishears some names, so a small correction dictionary fixes them afterwards, and I can teach her new corrections by voice.
• Thinking: gemma4 (26B) through Ollama. It is multimodal, so the same model can also read a screenshot and point at things on the screen.
• Speaking: Kokoro-82M, a small text-to-speech model running in its own Python server.
• Knowing who is talking: a tiny speaker-recognition model on the CPU, so she only acts on my voice and not on the TV or someone else in the room.
• Remembering: nomic-embed-text turns what was said into vectors for the memory (more on that below).

How a turn works

The project is a TypeScript monorepo with a package for each part: core, speech to text, the language model, text to speech, tools and the overlay. The parts never call each other directly. Everything goes through one typed event bus: the speech package announces what it heard, the core decides what to do with it, the language model answers, and the voice package speaks. That makes it easy to swap a model or test one part with the others mocked.

The language model works in a loop of up to eight steps. In each step it can call tools, like searching files, reading the screen, searching the web through a local SearXNG or taking notes. There are about 100 of them. After anything that changes something, like writing, moving or deleting a file, the loop has to check the result before it is allowed to say it is done. Bigger jobs can be started as a background "mission", where she plans the steps and reports back when she is finished.

You can interrupt her at any time, by saying "stop" or with a hotkey. That cancels the answer and cuts the voice off mid-sentence.

Memory

Every turn of the conversation is saved in a local SQLite database. On top of that, the important parts are stored as vectors in ChromaDB, so she can find things by meaning instead of exact words, like "what was I working on yesterday?". If the vector database is not running, she falls back to a plain keyword search, so she never loses her memory completely.

The model can hold a long context, but not forever, so the conversation she sees is kept within a token budget. Older parts are folded into a running summary instead of being cut off.

Safety

• Commands that could do real damage are blocked outright.
• When she asks for a yes or no, the answer has to be heard clearly. If she is unsure, it counts as a no.
• She can draft an e-mail, but she can never send one.
• Her personality is a config file, not code, so it can be tuned without touching the program.

The overlay

Grace has no window of her own. The only thing on screen is a small overlay at the edge of the screen that shows her state: idle, listening, thinking or speaking, along with what she heard and what she is saying. Clicks go straight through it, so it never gets in the way. The picture above is the real overlay, fed with a sample conversation in Danish.

She also has a few modes for different situations, such as a discreet mode, a brainstorm mode, a meeting mode where she mostly listens, and a field-notes mode for dictating notes.

It is far from done. The parts work and the full voice loop has worked in live tests, but an assistant that is always listening and actually useful day to day does not work yet.