Mechanics Behind Generative AI

Resources

• Transformers, visually explained — 3Blue1Brown connects text completion, embeddings, attention, and next-token probabilities with inspectable visual models. Use it to introduce generative AI and learned associations without implying that fluent output is verified knowledge. https://www.3blue1brown.com/lessons/gpt/

• Attention Is All You Need — The original Transformer paper supplies the mechanics behind self-attention, positional information, and parallel training. Pair it with the visual explanation, distinguishing the original encoder–decoder architecture from contemporary decoder-only language models. https://arxiv.org/abs/1706.03762

• Tokenization you can inspect — OpenAI’s tiktoken implementation makes byte-pair encoding concrete and testable. Useful for showing why characters, words, context-window budgets, and output-token limits are different units. https://github.com/openai/tiktoken

• How models learn to follow instructions — The InstructGPT paper separates pretrained completion behavior from supervised instruction tuning and reinforcement learning from human feedback. It grounds the distinction between base knowledge, instruction following, and preference-shaped responses. https://arxiv.org/abs/2203.02155

• Building effective agents — Anthropic distinguishes models augmented with retrieval and tools from predefined workflows and model-directed agents. Use its routing and specialization patterns to explain what a harness adds, where custom tools execute, and why mixing models is an engineering decision rather than a capability of prompting alone. https://www.anthropic.com/engineering/building-effective-agents

• OpenAI’s model catalog — Official model entries expose capabilities, supported interactions, context windows, output limits, and knowledge cutoffs. Use these live specifications to teach model selection and configuration rather than embedding a soon-stale model leaderboard in the lesson. https://developers.openai.com/api/docs/models

• Gemini’s model catalog — Google’s model documentation provides a second provider’s vocabulary for modalities, capability support, versions, and token limits. Compare specific entries with OpenAI’s catalog to show that settings and feature compatibility are not interchangeable across providers. https://ai.google.dev/gemini-api/docs/models

• The economics of a model call — Anthropic’s pricing reference distinguishes model tiers and charges associated with input, output, caching, and tools. Use it to reason about specialization and model mixing using total workflow cost, not just the advertised input-token price. https://platform.claude.com/docs/en/about-claude/pricing

• Structured output is a contract, not truth — OpenAI documents schema-constrained responses, supported schema features, and handling refusals or incomplete output. This anchors the difference between ordinary text generation, parseable JSON, schema compliance, and factual correctness. https://developers.openai.com/api/docs/guides/structured-outputs

• The function-calling round trip — OpenAI’s guide shows tool definitions, generated arguments, application-side execution, and returning results to the model. Use it to demystify custom tools: a model requests an action, while software must validate and perform it. https://developers.openai.com/api/docs/guides/function-calling

• Built-in tools versus external tools — The OpenAI tools guide maps hosted capabilities and externally connected tools onto the response interface. It gives teachers a concrete way to separate text prediction from the harness and runtime that provide search, execution, or other actions. https://developers.openai.com/api/docs/guides/tools

• Why language models hallucinate — This research paper analyzes how training and evaluation can reward plausible guesses over admitting uncertainty. Useful for explaining confabulation and the illusion of reliable recall without claiming that a larger model or a better prompt eliminates unsupported assertions. https://arxiv.org/abs/2509.04664

• Inside learned associations and reasoning — Anthropic’s circuit-tracing study investigates internal features, conceptual associations, planning, and cases where verbal explanations do not faithfully describe computation. Use it to ground discussion of latent representations; it does not establish the syllabus term “J-space,” which needs an explicit author definition before teaching. https://transformer-circuits.pub/2025/attribution-graphs/biology.html

• Self-ask and compositional reasoning — The original self-ask paper studies explicitly generating intermediate questions and optionally answering them through search. It supplies a concrete meaning for self-querying while separating prompted reasoning traces from evidence about internal computation. https://arxiv.org/abs/2210.03350