Video brief · S01E02

Steering Models Through Context

Brief in progress — review the script and demo before recording.

The goal of this video is to show the mechanics behind steering LLM behavior through what we can see (thread / prompts / settings) and what we cannot see (generalized harness mechanics).

  • Native Capabilities: what a model’s native capabilities are: text/image/audio/video processing and generation. The student needs to know where the line is between what the model does and where tooling comes into play.
  • Harness Layer: what the harness layer brings in terms of the UI: threads, attachments, settings (model, reasoning), and configuration ([].md files). This can be shown across multiple harnesses, focusing on similarities and only the most important features. This part is not about explaining features, but showing which of the things we see come from the harness.
  • Environment: what the model sees because of the harness, such as system / developer messages, attachments, references, and cache-friendly developer message injection that makes the agent aware of its environment.
  • Interactions: how the harness uses native tools, programmatic tool use, and the filesystem, how it accesses skills, and what those skills look like from the model’s perspective.
  • Collaboration: a brief overview of how subagents work, how agents can communicate, branch, and fork, and where this is useful in practice. This is not about detailed mechanics, but a general overview and some practical applications.

We may want to showcase scenarios such as:

  1. Prompting with an image taken autonomously by another agent.
  2. Presenting similarities between all the agentic tools, not just coding agents.
  3. Asking 3-4 harnesses about the structure of their context, comparing how Claude Code / Codex / Pi differ, and showcasing how they inject skills and agents into the context.
  4. Asking Pi to spawn another instance in herdr to get a worker and comparing it with Claude Code / Codex native subagents.