Tools

Inside Socratic Tutor: AI That Questions Before Answering

Jane Street's open-source Socratic Tutor uses follow-up questions to pinpoint gaps in a student's reasoning. Its Agent Skill design shows why effective tutoring depends on strategy switching, state management, and evaluation—not merely prompting.

Font size

Jane Street has released Socratic Tutor in its agent_skills repository on GitHub. Rather than immediately revealing the answer, it uses successive questions to identify gaps in the student's reasoning. For teams building educational products, that is more worth studying than adding another layer of “Socratic prompting.”

Inside Socratic Tutor: AI That Questions Before Answering

Jane Street published Socratic Tutor in its agent_skills repository on GitHub. The related discussion reached the Hacker News front page, earning 551 points and 129 comments.

What caught developers' attention was not another chatbot with better question-answering skills, but an AI Agent willing to withhold the answer and keep probing the student's reasoning.

Socratic Tutor is lightweight: a reusable SKILL.md file. Yet it raises a problem educational products must take seriously: students often need to locate their own sticking points rather than receive an answer.

Behind 551 Points, Developers Are Buying the Process

The Hacker News response can create the impression that this is just another “Agent hits the front page” project.

Yet 551 points and 129 comments suggest developers are focused on something else: how to transform models from answer generators into active guides within the learning process.

Most question-answering products optimize for faster responses, higher accuracy, and more direct outcomes. A student asks “Why does this happen?” and the system returns an explanation. The student then asks “How do I do it?” and the system supplies the steps. This is efficient, but it can replace the part of the work the student should be doing.

Socratic Tutor takes the opposite approach. Instead of answering directly, it uses follow-up questions to expose reasoning gaps, clarify concepts, and guide the student toward an answer.

This reflects a clear product direction: educational AI is shifting from generating answers to supporting the learner's reasoning.

That said, 551 points do not prove the approach improves learning outcomes. Hacker News readers may simply like the interaction model or be interested in Jane Street's engineering practices. Its real instructional value would require more rigorous controlled experiments.

How It Differs From a Conventional Chatbot

A conventional chatbot follows a basic loop:

User asks a question → Retrieve information → Construct an answer → Return result

Socratic Tutor follows a different loop:

Student responds → Check concepts and reasoning → Identify the gap → Ask the next question

The difference goes beyond adding “use the Socratic method” to a prompt. It changes the system's objective when interacting with the learner.

Successive Follow-Up Questions

A single question is merely a prompt; successive questions create a diagnosis. If a student says, “The result holds because the function is monotonic,” the Tutor must continue by asking: What does “monotonic” mean here? Over what interval? Is it strictly increasing?

Each question must relate to the reasoning at hand. It should not ask generically, “What else would you like to know?” or repeat a question to prolong the conversation when the issue has already been resolved.

Concept Clarification

Many errors are not computational mistakes but instances of mixed-up concepts. A student may conflate correlation with causation, treat a sample mean as a population mean, or overlook boundary conditions.

Effective follow-up questions force students to articulate the concepts and conditions they are using. If a student's explanation is contradictory, the Tutor should identify the contradiction before introducing the correct terminology.

Exposing Reasoning Gaps

Traditional RAG question-answering templates focus on whether relevant documents were retrieved and whether the answer cites its sources. Socratic Tutor must answer another question: Which step is missing between the student's premises and conclusion?

Its ideal output is therefore not a polished answer but a sequence of diagnostically useful questions:

  • Which assumption are you relying on?
  • Would the conclusion still hold if that condition were false?
  • Does your method use the same definition as your example?

This means the system must maintain conversational state: what the student has already explained, what they have misunderstood, and which unresolved step remains.

A SKILL.md Can Define an Agent's Teaching Persona

Socratic Tutor is an instructional Agent Skill released by Jane Street in its GitHub agent_skills repository. It defines Agent behavior through a reusable SKILL.md file rather than hard-coding the teaching flow into application code.

The original instructions are available here:

https://github.com/janestreet/agent_skills/blob/master/socratic-tutor/SKILL.md

This approach has practical value for Agent frameworks. A teaching strategy becomes a readable, reviewable, reusable behavioral specification rather than a series of conditional statements in a Python file.

It can be understood as an Agent's “role configuration layer.” Product teams can still connect models, tools, state management, and interfaces at the outer layer, while SKILL.md describes the instructional method in the middle.

It provides at least four benefits:

  1. Reviewable behavior. Product managers, teachers, and evaluators can read the teaching rules directly.
  2. Replaceable strategies. The same Agent framework can load different Skills.
  3. Lower experimental costs. Changes to instructional logic do not require rewriting application code.
  4. Cross-runtime reuse. Any target Agent that can read a Markdown specification can adopt similar behavior.

A Markdown file is not magic, however. It works well for expressing “what to probe” and “when to change strategies,” but it is unsuitable for implementing a complete state machine, tool permissions, or an evaluation loop.

If a Skill merely says “keep asking until the student understands,” the model may mechanically repeat questions or pretend to have diagnosed a misconception when it lacks context. The real engineering work remains in the surrounding system: storing student responses, identifying concepts already covered, and selecting an appropriate method for the situation.

Too Many Questions Can Wear Students Down

Socratic teaching has an obvious risk: turning “supporting thought” into “refusing to help.”

Real learning situations vary:

  • A student encountering a concept for the first time may not know how to frame the question.
  • A student reviewing for an exam may need a quick check of the key steps.
  • A student who has already tried several times may need only a minimal hint.
  • A user explicitly requesting verification of a final answer needs confirmation, not another round of questioning.

The most valuable aspect of Socratic Tutor should therefore not be “never answer directly,” but “know when not to answer directly.”

A more reliable strategy would include at least four levels:

Diagnostic question → Targeted hint → Example or analogy → Give the answer with an explanation

The triggers for switching levels should also be concrete. Repeated answers containing the same error, an explicit request for a hint, limited time remaining, or missing prerequisite knowledge should all lower the level of questioning.

After a student becomes frustrated, the Agent should also allow an exit. Otherwise, “not replacing the student's thinking” can easily become “raising the cost of giving up.”

The Real Value Lies Beyond Prompting

Evaluating an instructional AI Agent by final-answer accuracy alone is insufficient.

Many products already solve standard questions well. Even if the model never asks a follow-up question and simply produces the correct answer, its final-answer accuracy will be high. But such a system cannot distinguish among these cases:

  • The student genuinely understands the material.
  • The student merely copied the AI's intermediate steps.
  • The AI happened not to trigger the student's typical misconception.

The following metrics deserve more attention:

  1. Misdiagnosis rate: When the Agent claims the student has a particular misconception, does it actually exist?
  2. Follow-up hit rate: Does each question target a specific error that affects the solution?
  3. Prompt escalation accuracy: When a hint is needed, is the assistance small enough to preserve discovery but effective enough to move the student forward?
  4. Independent solution time: After tutoring, how long does the student need to solve a similar problem independently?
  5. Transfer after prompting: Can the student apply the method to a new problem rather than repeat the previous answer?

Reducing the time students need to solve problems independently is closer to the product goal than measuring one-time answer accuracy. It is also harder to measure because evaluations must control for problem difficulty, prior knowledge, tutoring duration, and exit conditions.

One practical evaluation method would assign the same problem to several instructional strategies: direct answers, a fixed follow-up script, and Socratic Tutor. Students who do not know their group assignment could then be assigned randomly, after which researchers could compare independent solution time, requests for hints, and error patterns.

The model version must also remain fixed. Otherwise, changes in reasoning ability introduced by a single upgrade could overshadow the effect of the instructional strategy itself.

Editorial Perspective

Socratic Tutor is worth examining not because it proves that “AI teaches better than teachers,” but because it shifts the product objective from “producing answers faster” to “exposing errors earlier.” This changes how instructional AI Agents are optimized: final-answer accuracy is no longer enough, while independent solution time and error transfer become more important.

The next wave of products will likely differentiate themselves less through larger models than through state management, strategy switching, and outcome evaluation. Knowing when to ask a question is harder than knowing how to ask it.

Unlike many earlier AI tutoring features, this project does more than replace the interface with Socratic language. It attempts to codify a method of intervention through a reusable Skill. Even so, 551 points and 129 comments show developer interest, not improved learning outcomes.

Rigorous controlled experiments are still needed. Until independent solution time and transfer have been validated, Socratic Tutor is better treated as a teaching-design reference than an established product paradigm.

Source: https://github.com/janestreet/agent_skills/blob/master/socratic-tutor/SKILL.md