The Socratic Tutor: an AI tutor that asks instead of answers

by

The problem

Teaching programming to a roomful of students is, before anything else, an attention-allocation problem. Twenty students get stuck on twenty different things at the same moment, and the instructor can only be in one place at once. The pressure to relieve that by handing out the worked solutions is constant — but doing so quietly undermines the thing you were trying to teach. Reading the solution feels like progress; it tells the student what the answer is without forcing them through the discomfort that builds the skill. Most students, given the choice between struggling for twenty minutes and copying a working snippet, copy. They are not wrong to: the immediate reward is real. But the productive struggle is the part that sticks.

The other tools available to a student in 2026 push in the same direction. A general-purpose chat assistant, asked “how do I solve this?”, will gladly produce a clean, runnable solution — often within seconds. From the student’s point of view, that is indistinguishable from getting the answer from the instructor. The fact that an AI happens to be the one handing it over does not change what was lost.

What would help, instead, is a tutor that has seen the reference solution but is constrained — at a level the student cannot override — to never give it away. A tutor that asks where you are stuck, points at the missing concept, and nudges you one step further, but stops short of completing the task. That is what this extension is.

The idea

The Socratic Tutor is a chat participant in VS Code. When a student opens the chat sidebar and writes @tutor I'm stuck, the extension:

  1. Reads a task identifier the student has marked in their notebook (e.g. #| task: lesson-1).
  2. Fetches the corresponding reference solution from a private GitHub repository that only the instructor controls.
  3. Passes the student’s current code, the conversation history, and the reference solution to the underlying language model — along with a system prompt that explicitly forbids reproducing the solution verbatim and lays out a multi-turn escalation policy.
  4. Streams the model’s reply back into the chat.

The student never sees the reference solution. The model uses it only as ground truth for diagnosing what the student is missing, choosing what to ask, and deciding how strong a hint to give.

What it looks like

Student: @tutor my model won't converge — what am I doing wrong?

Tutor: Before we look at the model, what assumption is your formula making about which variables are part of the network and which are node attributes? Try running summary() on your network object first and tell me what you see.

Student: (pastes summary output)

Tutor: Good — notice that your tie variable is being read as a node attribute, not an edge. Your formula is then asking the estimator to predict something that isn’t in the data. Look at the line where you build the network object; one argument is in the wrong position.

The tutor’s responses get progressively more concrete across turns. The first response usually contains a guiding question and a small hint. By the third response on the same problem, the student gets a one- or two-line illustrative snippet — but never the full solution.

How it works under the hood

The whole extension is around 350 lines of TypeScript. The interesting parts:

  • Task ID detection. The extension scans the active editor for a marker matching #| task: <id> (Quarto’s cell-attribute syntax), or accepts a /task <id> chat command. The ID has two parts separated by the last -: a notebook prefix (e.g. lesson) and a task number (e.g. 1).
  • Solution lookup. The notebook prefix tells the extension which file to fetch from your solutions repo — e.g. lesson.qmd. The full file is downloaded via the GitHub Contents API, cached per-session, and parsed.
  • Solution extraction. Inside the notebook, the extension finds the heading whose attribute block contains {lesson-1}, then scans forward for the next Quarto callout titled "Solution": ::: {.callout-tip title="Solution"} ... :::. Everything inside that callout is pulled out (respecting nested fenced divs) and used as the reference solution.
  • The system prompt. This is the heart of the pedagogy. It instructs the model to diagnose before answering, prefer questions over scaffolds and scaffolds over snippets, escalate over multiple turns, never reproduce the solution verbatim, and never claim correctness it can’t verify. The full prompt lives in src/tutor-system-prompt.md and is loaded once when the extension activates; instructors are encouraged to edit it for their own course’s tone.

Setting it up for your course

This is the part that turns the rest into something you can actually use.

Step 1 — Prepare your solutions repo

Create a private GitHub repository, e.g. your-username/course-solutions. Inside it, make a folder (the default name is notebook-solutions) and add one Quarto file per “lesson” or “notebook” in your course.

A working starter file is included in the public repo at templates/solution-template.qmd — copy it into your solutions repo, rename it (e.g. r-lab.qmd), and replace the example exercises with your own. The minimal structure it demonstrates is:

# R Lab: Loops and Conditionals

## Task 1 `{r-lab-1}`

Write a function that sums every other element of a list.

::: {.callout-caution collapse="true" title="Solution"}

```r
sum_alternate <- function(x) {
  sum(x[seq(1, length(x), by = 2)])
}
```

:::

Two non-obvious requirements:

  • The heading must contain the task ID inside {curly braces}. The tutor matches on the literal pattern {r-lab-1} somewhere on the heading line.
  • The reference solution must be wrapped in a ::: {.callout-* title="Solution"} ... ::: block. Any callout type works (callout-tip, callout-note, callout-caution) — only the title="Solution" matters.

Step 2 — Create a GitHub PAT

Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token.

  • Resource owner: your GitHub user.
  • Repository access: Only selected repositories → pick your solutions repo, and nothing else.
  • Repository permissions: Contents = Read-only. Leave everything else at “No access”.

Set an expiration date that aligns with the end of your semester so the token rotates naturally.

A note on security: if you distribute the .vsix to students, do not hardcode this token into the source. Students can unzip the .vsix and read it out. The whole point of the SecretStorage flow described below is that each instructor’s token lives only on their own machine.

Step 3 — Install the extension

Either:

  • Download the latest socratic-tutor-x.y.z.vsix from the public repo’s Releases page, or
  • Clone the repo, run npm install, then npx vsce package to build it yourself.

Then in VS Code: Extensions panel → ... menu → Install from VSIX… → pick the file. Reload the window.

Step 4 — Configure

Two one-time steps:

  1. Open the Command Palette (Ctrl+Shift+P) and run Socratic Tutor: Set GitHub Token. Paste the PAT from step 2. It is stored in your OS keychain via VS Code’s SecretStorage API — never in any settings file.
  2. Open Settings (Ctrl+,), search for “Socratic Tutor”, and fill in:
    • Repo Owner → your GitHub username
    • Repo Name → course-solutions (or whatever you named it)
    • Solutions Path → notebook-solutions (or empty if your files are at the repo root)
    • File Extension → qmd (the default; change if you use .md or something else)

Verify by opening a chat with @tutor and asking @tutor test connection. You should see a count of how many solution notebooks it found.

Step 5 — Tell your students how to mark tasks

In their working notebooks — not the solution file, where the tutor identifies tasks by the {taskId} in the heading — students mark the current task with a Quarto cell directive at the top of a code chunk:

#| task: r-lab-1

# their code here

Or, in the chat, they can type /task r-lab-1 to set the current task explicitly.

That is the whole setup.

Customizing the tutor for your domain

The default system prompt is intentionally generic — it talks about “programming exercises” and avoids any reference to specific languages or topics. For your course, you will almost certainly want to make it more specific: name the language, mention the libraries students should reach for first, swap in Socratic example questions that use your domain’s vocabulary.

The prompt is a plain markdown file at src/tutor-system-prompt.md, with ${fileName}, ${taskId}, and ${solution} placeholders that get substituted at request time. Edit the file, rebuild the .vsix with npx vsce package, and distribute the new version. There is no separate configuration for the prompt because, in my experience, getting it right for a course requires iterating on real student interactions — a settings UI would only encourage shallow tweaks.

Try it

The code is at github.com/benrosche/socratic-tutor-public. If you adopt it for your course, I’d be interested to hear how it goes — both the wins and the places where the model’s hint quality breaks down.