Coding Challenge Session: Local Browser Conversational Chatbot (STT, TTS, and more?)

Prioritize local open-source over cloud AI

Shiffman emphasizes running models locally on consumer hardware rather than using closed cloud-based systems from big tech companies to maintain data privacy and agency.

AI for artistic expression, not productivity

The goal is demystifying AI through creative coding, social commentary, and weird art projects rather than building useful assistants.

Simple brains beat complex LLMs

You don't need large language models—Markov chains, context-free grammars, or pattern matching can power creative chatbots effectively.

Speech recognition with Whisper

Uses OpenAI's open-source Whisper model (Tiny variant optimized for English) converted to ONNX format for browser compatibility via Transformers.js.

Lightweight voice synthesis

Implements Kokoro TTS, a recent small-footprint text-to-speech model from Hugging Face that runs fast on local hardware without cloud APIs.

Browser-based ML stack

Leverages the Hugging Face Transformers.js library imported via CDN to run ML pipelines directly in p5.js sketches using modern async/await patterns.

Push-to-talk interface

Creates a minimal UI where holding the mouse button starts recording (red) and releasing triggers transcription (green).

Model transparency checks

Stresses the importance of reading model cards and cites Margaret Mitchell's framework for transparent model reporting before implementation.

ONNX standard format

Uses Open Neural Network Exchange (ONNX) format to ensure model weights are compatible with JavaScript and browser-based inference.