Two Products, One Engine:
Realtime Diarization & Local Voice Input

Executive Summary

Both products are realisable today with open-source, local-first components, no cloud dependency, no API fees. The two projects share a single AI foundation (speech → text → meaning) and are technically synergistic: the voice input method is essentially a single-speaker special case of the dialogue recorder. Building them as a shared "voice-AI core" with two thin product surfaces is the highest-leverage architecture.

Research Method

Analysis grounded in live open-source reconnaissance (Agent-Reach / Exa semantic search + GitHub/README extraction, August 2026) plus the local stt-transcription skill reference. Key sources: Diart, pyannote.audio 3.x, sherpa-onnx, OpenTypeless, AriaType, and speaker-to-name LLM literature (arXiv 2509.15082). Figures marked as of 2026-08.

---

Part 1: Project 1: Dialogue Recorder & Realtime Analyzer

Goal: record a multi-person conversation in real time; continuously identify who is speaking by voice characteristics; then assign names to those voices based on content context, all with lightweight, preferably local, models.

1.1 Decomposing the problem

The task splits into four distinct capabilities, each with a mature open-source answer:

1.2 The realtime engines: Diart (the right choice)

juanmc2005/diart is a Python framework purpose-built for real-time speaker diarization (MIT, ~2k★). It fuses a speaker-segmentation model and a speaker-embedding model into an incremental clustering pipeline that becomes more accurate as the conversation progresses, exactly the online behavior Project 1 requires. Configurable latency (as low as ~500 ms) and native WebSocket serving make it the natural engine.

# Diart streaming speaker diarization (core loop)
from diart import OnlineSpeakerDiarization
from diart.sources import MicrophoneAudioSource
from diart.inference import RealTimeInference
from diart.sinks import RTTMWriter
import diart.inference as rec

pipeline = OnlineSpeakerDiarization.from_pretrained("diart/speaker-diarization-xenova")
config = pipeline.config
async def on_stream(wav):
    for turn, _, speaker in pipeline(wav):
        yield turn, speaker  # (start,end) → SPK_x, live

# Serve over WebSocket for a recorder + live UI
inference = RealTimeInference(pipeline, MicrophoneAudioSource(config.sample_rate))

1.3 Name assignment from content: the "context" trick

Diarization gives anonymous clusters (SPK_0). Assigning real names is a content-inference problem, solved in the research literature (e.g. arXiv 2509.15082 "Identity-Aware LLM Refinement of Speaker Diarization") and by AssemblyAI's LLM workflows: feed the diarized transcript + a roster of expected names into a local LLM, which maps clusters to identities using linguistic context, "as Mike said," self-introductions, topic affinity.

Local model choice: any instruct LLM ≥7B (Llama-3.1-8B, Qwen-2.5-7B, Mistral-7B) running via llama.cpp/Ollama is enough, the task is a short structured reasoning pass over segments, not heavy generation. Prompt: give diarized transcript + name roster, ask for a cluster→name map with confidence.

1.4 Recommended stack (all local, lightweight)

1.5 Build plan: Project 1 (phases)

Part 2: Project 2: Voice Input Method (Typeless-style, local)

Goal: a global hotkey → speak naturally → polished text injected into whatever app you're in. Like Typeless, but powered by a lightweight local model (no cloud).

2.1 State of the open-source market

The space already has two strong references, one cloud-BYOK, one local-first:

Key insight: AriaType already proves the exact architecture you want, a Tauri desktop app driving a llama.cpp/ONNX sidecar for fully local speech-to-text + LLM "polish." Your build is a focused, product-polished iteration on this pattern (ideally landing as your own opinionated app, or a clean fork), not a from-scratch research problem.

2.2 Architecture (local-first)

Global hotkey ─▶ microphone capture (Rust cpal)
                     │
                     ▼
              [ Local STT: SenseVoice / Whisper / sherpa-onnx ]  ── text
                     │
                     ▼
              [ Local LLM "polish": llama.cpp sidecar (8B) ]  ── clean, punctuated text
                     │
                     ▼
              Inject into focused app (Windows SendInput / macOS CGEvent)
                     │
              Optional: per-app context & custom templates

2.3 Build plan: Project 2 (phases)

Part 3: Shared Core & Cross-Cutting Strategy

The two products are one engine with two surfaces. Recommene building a shared voice-AI core:

Build order recommendation

Risks & mitigations

Deep analysis grounded in Agent-Reach / Exa / GitHub reconnaissance, August 2026. Model names, star counts and licenses as of research date; verify before committing.