LUCILLE

Not a chat window. A presence on the desk.

The design constraint that shaped everything: Lucille is a voice-first appliance, not an app you open. She boots when the Mac mini boots, sits full-screen on a small display, listens for her name, and answers out loud in one to three sentences. Every architectural decision downstream — routing, streaming speech, barge-in, the trailing-text filter — exists because a reply that is read aloud has completely different requirements than a reply you read.

• Local-first by default. She was built to run for free and offline — Ollama for thinking, Apple/Kokoro for voice. Claude is an upgrade, not a dependency.
• Side-loaded, deliberately. App Store sandbox rules effectively forbid always-listening wake words, so she ships as an Xcode build dropped into /Applications and added to Login Items.
• Capabilities grow through tools, not through her core. New skills arrive as client-side Swift tools, bridge endpoints, or hosted MCP servers listed in a plist.

Apple frameworks on the outside, two model runtimes on the inside.

From air pressure to an answer, and back out loud.

One AssistantViewModel on the main actor owns all state; every service is a leaf it wires together with closures and Combine publishers. State changes fan out to the face, the menu bar, and any connected phone in the same assignment.

Four trees run her: route, turn loop, inbox, dark web.

The routing heuristic is the single most consequential piece of logic in the app — it decides cost, latency, and whether she can use tools at all. It was later extracted into a pure RouteDecision function precisely so it could be unit-tested in isolation.

// Hybrid policy: keep trivial chit-chat on the fast local model, and
// send anything substantive to Claude. Default is cloud — local is
// the exception, not the rule.
private func chooseRoute(for prompt: String, mode: RoutingMode) -> RoutingMode {
    if mode != .auto { return mode }

    let lower = prompt.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
    let wordCount = lower.split(separator: " ").count

    if trivial.contains(lower) { return .local }

    // Very short, content-free utterances stay local — unless they're a
    // question (starts with a question word), which goes to Claude.
    let firstWord = lower.split(separator: " ").first.map(String.init) ?? ""
    if wordCount <= 2 && !questionWords.contains(firstWord) {
        return .local
    }

    return .cloud
}

She answers before she has the answer.

A cloud turn with thinking and web search can run twenty seconds. Silence for twenty seconds reads as broken. So the cloud task is kicked off first, and while it runs the local model generates a two-to-six word holding phrase that is spoken immediately — "Checking your calendar." The acknowledgment is deliberately never written to memory, so it can't pollute later context.

Seventeen tools across four execution surfaces.

The tool array is assembled per request. Anthropic's own tools run server-side; the rest execute in Swift, mostly by forwarding to the Google bridge on :8770. Hosted MCP servers can be added by editing a plist — every server is auto-paired with a toolset, with an optional allowlist so a destructive tool never quietly enters a voice assistant's reach.

The system prompt does a surprising amount of the engineering here: it forces the style guide to be loaded first, forbids stopping mid-task to ask permission, and bans claiming an action succeeded without a tool result confirming it.

Her most useful tool is a five-call chain she is forbidden to shortcut.

“Draft a client alert on the Delta Dental NYDFS consent order” is the single request that pushed this whole app from toy to tool. It is also the request that broke the most, because it needs research, style fidelity, deterministic document rendering and email — in one unbroken turn, while she is talking to you.

What it took to make this work at all

• The 4096-token starvation bug: research plus a full draft blew past the ceiling and the turn hit max_tokens mid-work, so create_client_alert was never reached. The ceiling is now 16,000 — spent on thinking and orchestration, while the spoken reply stays capped at 1–3 sentences.
• The container-id 400: web_search runs server-side code execution, which spins up a container. Once a turn uses it, every follow-up request must reference that container by id. Research and document creation span a turn boundary, so the id has to be captured and threaded back — this was the exact bug that broke client alerts.
• Fourteen turns of headroom: maxTurns is 14 specifically because multi-step skills like this one need room to search, read, draft and render without the loop cutting out.
• No permission-asking: the system prompt forbids saying “I'm on it” and stopping, and forbids asking for a topic that was already named. “The Delta Dental NYDFS consent order” is the topic.
• Never a false success: she may only report an action succeeded when a tool result confirms it — the same rule that governs calendar invites and Meet links.

The same substrate carries the staged NDAA program — trackers, comparison charts, one-pagers, implementation updates — because the skill is a set of resources plus two tools, not a hardcoded feature. Adding a new practice area means adding resources, not shipping a new build of the app.

When she reaches into Abyssal, her eyes turn red.

Abyssal is the other half of this stack: a multi-agent dark web research tool that answers a plain-English question by planning subqueries, fanning them across Tor, and returning structured findings instead of pages. Wiring it into Lucille as an MCP server gives her a capability no consumer assistant has — and that deserved a visible tell. Red visor means a Tor circuit is open somewhere on her behalf.

The parts that are actually hard

• Freshness vs. politeness: a warm cache answers instantly, but threat-intel questions are about today. The API exposes an explicit fresh-crawl option, and SSE emits live-fetch events — page fetches, extracted victims, CAPTCHA hits — so you can watch a live sweep instead of staring at a spinner.
• Mirrors move constantly: the YAML registry carries aliases, active flags, mirror lists, CAPTCHA and JavaScript hints, and per-site metadata, so adding a source is a pull request against a config file rather than a code change. Mirror health is tracked, because the primary is down more often than not.
• CAPTCHAs are a policy question, not just a technical one: handling is an explicit solver subsystem — manual by default, with optional 2Captcha, Anti-Captcha and CapMonster providers. SiteAgent detects the challenge, pauses into a pending-solve queue, injects the returned token, refetches, and escalates on failure.
• The same victim appears five times: SimHash near-duplicate collapse plus per-source confidence weights are what turn a pile of findings into a count you can put in a memo.
• Answers have to survive the site: the append-only SQLite victim store, screenshots and OCR, and STIX/CSV/JSON export exist so a finding is still provable after the leak site rotates or goes dark.

Design note: this is the only place in the entire interface where color changes meaning. Everything else is phosphor white on near-black, so red reads instantly across the room — and it clears itself the moment the last worker returns, because a mode you can forget you left on is a bad mode.

Yes — but the arrow points both ways.

Lucille already speaks MCP: servers are declared in her MCPServers plist, each auto-paired with a toolset and an optional allowed_tools list. So adding Perplexity is a config entry, not a feature build. The honest nuance is that Computer is an MCP client — it consumes servers rather than exposing itself as one — so the two systems meet in the middle, from opposite directions.

Both directions follow the same principle the rest of the build follows: capability arrives as configuration with an explicit allowlist, so nothing destructive ever enters a voice assistant's reach by accident.

Six states, six expressions, one glowing visor.

The face went green CRT → warm phosphor white → bigger, friendlier eyes → explicitly EVE-from-WALL·E, because the earlier robotic version read as unsettling. The final design carries emotion the way EVE does: through tilt and vertical scale, not through added features. There is no mouth — only a 24-bin waveform driven by real audio metering.

A JSONL log, and the discipline to keep it clean.

The iPhone is a handset, not a second brain.

The Mac keeps the brain, memory, and identity. The phone app mirrors her face live and acts as push-to-talk: it transcribes on the phone, sends only text, and speaks her reply back locally. Just small JSON messages cross the network, so it stays responsive on cellular.

• Transport: newline-delimited JSON over raw TCP on port 47771 — face state down, commands up.
• Discovery: Bonjour (_lucille-face._tcp) on the same Wi-Fi; a Tailscale address anywhere else.
• Why not iCloud: CloudKit is a paid-account capability. Tailscale keeps it free, encrypted, and with nothing publicly exposed.
• Robustness: per-peer inbound buffers, since one TCP read can hold a partial line or three whole ones; auto-reconnect on drop.
• Queueing: remote prompts are de-duplicated by UUID and only drain while she is idle, exactly like inbox tasks.

Most of the good code here is scar tissue.

A weekend prototype, then a day of hourly passes.

The interfaces are already in place; only the guts change.

• Vector memory: swap MemoryStore for SQLite + sqlite-vec behind the existing append/recall pair.
• Skills as MCP servers: keep style and instruction resources separate from deterministic generation, so new practice-area skills need no core changes.
• Local file recall: Spotlight via NSMetadataQuery, exposed as one more Swift tool.
• Home control: HomeKit through HMHomeManager, or Home Assistant's REST API as a bridge endpoint.
• Honest builds: regenerate build.sh from the project file, so the CLI path can never drift from Xcode again.