Mechanics Behind Generative AI

Available materials

Optional reading, documentation, and tools for exploring the topics in this lesson. These include the lesson’s external references and selected further reading from the research notes.

LLM mechanics and tokenization

• Let’s build the GPT Tokenizer — A video walkthrough by Andrej Karpathy showing how tokenizers turn text into tokens and how to build one.
• What Is ChatGPT Doing … and Why Does It Work? — Stephen Wolfram’s explanation of text generation, neural networks, and next-token prediction.
• Tiktokenizer — An interactive tool for inspecting how text is split into tokens and comparing token counts.
• OpenAI Model Spec: definitions — Definitions of messages, roles, tools, and conversation structure, supporting the lesson’s explanation of chat formats.
• Transformers, visually explained — A visual introduction from 3Blue1Brown to embeddings, attention, and the mechanics of next-token prediction.

Reliability and reasoning

• Defeating Nondeterminism in LLM Inference — Thinking Machines explains why identical inputs can produce different outputs and how inference implementation affects reproducibility.
• Why language models hallucinate — OpenAI explains why models produce plausible but false answers and how training and evaluation can reward guessing over acknowledging uncertainty.
• Verbalizable Representations Form a Global Workspace in Language Models — Anthropic’s research on a shared workspace in language models, providing background for the lesson’s discussion of J-Space.
• Premise Order Matters — A research paper examining how changing the order of information in a prompt can affect reasoning performance.
• Self-ask and compositional reasoning — Research on answering complex questions by first generating and answering intermediate questions. Further reading for the self-querying section.

APIs, structured outputs, and code execution

• OpenAI API quickstart — A starting point for making API requests and generating responses from a model.
• Responses API reference — Reference documentation for the request and response format discussed in the lesson.
• Gemini Interactions API — Google’s documentation for its interaction API, useful for comparing how providers structure model interactions.
• Structured outputs — OpenAI’s guide to generating responses that follow a JSON Schema, including limitations and error handling.
• Function calling — OpenAI’s guide to defining tools, receiving generated arguments, executing functions, and returning results to the model.
• Cloudflare Code Mode — Documentation on letting models write code that combines tool calls and runs in a sandbox.
• Daytona — A sandbox platform for running code, relevant to the lesson’s discussion of executing agent-generated scripts.

Model comparison and access

• Artificial Analysis — Model comparisons covering capabilities, speed, and pricing. Useful when choosing models for different tasks.
• OpenRouter — A platform for accessing models from multiple providers through a shared API.

Coding tools and harnesses

• Cursor — An AI-assisted code editor for working with models and agents inside a development environment.
• Claude Code — Anthropic’s coding agent for working on software projects.
• Omp — A coding-agent harness included in the lesson’s discussion of alternatives for working with different models.
• Pi — An extensible coding-agent harness for customizing how you work with models and tools.
• T3 Code — An interface for working with coding agents on software projects.
• OpenCode — An open-source coding agent that supports working with different model providers.
• Superset — A workspace for coordinating coding agents and their work on a project.

Agent design and self-management

• Building effective agents — Anthropic’s practical guidance on workflows, agents, and combining models with tools and retrieval.
• OpenCode 2 — The article linked in the self-management section, discussing the design of OpenCode’s next iteration.

To prepare