Brief in progress — review the script and demo before recording.
The goal of this video is to explain the principles behind helping agents manage the knowledge they need to understand their work at both a narrow and a broad level, so they can make most decisions autonomously. Students also need to understand how agents perceive and discover knowledge.
- Native Capabilities: LLMs excel at many tasks, but bash, coding, and delegation are the ones labs focus on during training. We need to show how clever AI can be when using tools, but also how limited it is when navigating our personal context.
- Specifications: every AI interaction starts fresh, so agents need to be treated differently from humans. They need to understand not only what the task is about, but also where to find broader context, such as the project vision, the history of features (not just in git), and what’s planned. In other words, agents need both a wide and a narrow perspective on the task, while staying focused on what has to be done right now.
- Issues: agents can work for hours, and while models are getting better, agents can easily drift away from the vision and our expectations. This means scope, requirements, and progress tracking need to be properly managed during planning, during the actual work (harness setup), and afterward (external integrations).
- Global Context: the vision, style guide, business perspective, focus areas, etc. need to be surfaced, easily accessible to agents, and kept up to date. Some of these documents will come from us, but most can be inferred by agents from transcripts, forwarded messages, voice messages, or chats.
- Synchronization: documents drift over time, but AI can help us keep them organized, described, and up to date, either on a schedule or through direct work with an agent. This requires easy access for both us and the agent, so keeping the specs in Google Docs may not be a good idea.
- Exploration: the fact that we give agents specs in various forms does not mean they will automatically load them or keep them in context long enough to complete the task. Once we understand the mechanics of the harness, we can automatically inject specific instructions for a given type of worker or extend tool results to remind the agent of certain actions or steer it toward them, such as reloading the style guide.
We may want to showcase scenarios such as:
- Knowledge Base / Obsidian exploration where most documents can be found, but some won’t be. A good example would be an association such as
projects -> t3code. AI does not know about such a connection by default. - An example where the agent is asked to implement a feature twice: once with a specific vision.md provided, and once without it.
- An example in which the agent has access to specifications for implemented / active / done features and is tasked with an Onboarding Feature. It discovers existing features not only from the code, but also from the vision and the history of the project built so far, along with a planned feature that affects the shape of the onboarding.