The Socratic Tutor: an AI tutor that asks instead of answers

by

Teaching programming to a roomful of students is, before anything else, an attention-allocation problem. Twenty students get stuck on twenty different things at the same moment, and the instructor can only be in one place at once. The pressure to relieve that burden by handing out worked solutions is constant, but doing so quietly undermines the very thing you are trying to teach. The temptation is understandable on both sides. For the instructor, a worked solution clears the immediate bottleneck. For the student, reading it feels like progress: it reveals what the answer is without requiring the discomfort that builds the skill. Most students, given the choice between struggling for twenty minutes and copying a working snippet, will copy. They are not wrong to do so. The immediate reward is real. But the productive struggle is the part that sticks.

The other tools available to a student in 2026 push in the same direction. A general-purpose chat assistant, asked “how do I solve this?”, will gladly produce a clean, runnable solution — often within seconds. From the student’s point of view, that is indistinguishable from getting the answer from the instructor. The fact that an AI happens to be the one handing it over does not change what was lost.

What would help, instead, is a tutor that has seen the reference solution but is constrained to never give it away. A tutor that asks where you are stuck, points at the missing concept, and nudges you one step further, but stops short of completing the task. That is what this project is.

The idea

The tutor is a skill in Posit Assistant, backed by a small MCP server you run.

It is designed for Positron, and that is the integration that ships ready to use. Nothing about the server is Positron-specific, though — it is a standard MCP server, and any MCP-capable editor can talk to it. Using it in another IDE, below, has pointers.

When a student selects Tutor from the agent dropdown and writes “I’m stuck”:

  1. The skill reads a task identifier the student has marked in their notebook (e.g. #| task: r-lab-1).
  2. It calls get_task_context on your server, which looks up the reference solution and returns it as private model context — along with an escalation level.
  3. The model replies with a diagnosis, a question, or a small scaffold, calibrated to that level.
  4. The request is logged, so you can see where the class is stuck.

The student never sees the reference solution. The model uses it only as ground truth for diagnosing what is missing and deciding how strong a hint to give.

What an interaction with the AI tutor looks like in the chat pane

Student: my model won’t converge — what am I doing wrong?

Tutor: Before we look at the model, what assumption is your formula making about which variables are part of the network and which are node attributes? Try running summary() on your network object first and tell me what you see.

Student: (highlights summary output)

Tutor: Good — notice that your tie variable is being read as a node attribute, not an edge. Your formula is then asking the estimator to predict something that isn’t in the data. Look at the line where you build the network object; one argument is in the wrong position.

The tutor logic

The tutor answers the same exercise differently depending on how many times that student has already asked about it. The first ask gets a guiding question and a small hint. The second gets a stronger hint and a partial scaffold. The third and beyond get a small illustrative snippet, never the full solution. That count is the escalation level.

The count is stored in Postgres, keyed to the student and the exercise, rather than derived from the current conversation. It therefore carries across new chat windows, restarts, and reinstalls: a student on their fourth ask receives a fourth-ask answer in a chat window that is thirty seconds old. A counter kept in the client resets whenever the conversation does. The server exists for this counter and for the dashboard.

The dashboard

Each request writes a row: the student, the exercise, the level reached, the text typed, whether a reference solution was found, and the timestamp. The server renders those rows at /dashboard, behind a password, querying the database on each load rather than from a pre-rendered report.

The panels are:

  • Which exercises generate the most requests — request counts per exercise, drawn as bars.
  • Which exercises leave students stuck — for each exercise, the share of its askers who reached level 3 or higher.
  • Who is asking — requests per student, showing whether they are spread across the class or concentrated in a few.
  • When the work happens — requests per day, over the most recent 21 days with activity.
  • Asked about, but no solution loaded — exercises where students hit “not found”, meaning the solution notebook needs annotating and reloading. Shown only when there are any.
  • What students actually asked — the question text verbatim, most recent first.
  • Exercises nobody asks about — loaded, but never asked about.

How it works under the hood

  • Task ID detection. The skill reads the #| task: <id> marker from the active editor, which Posit Assistant attaches as context by default. In v0.1 this was a regex scan in extension code; now it is a sentence of prose in a markdown file.
  • Solution storage. A loader script parses your private solution notebooks and upserts them into Postgres. Solutions never live in the public repo or in the deployment, and updating them needs no redeploy.
  • Solution extraction. Inside a notebook, the parser finds the heading whose text contains {r-lab-1}, then scans forward for the next Quarto callout titled "Solution" and captures its body, respecting nested fenced divs. This code is moved verbatim from v0.1 — it never depended on VS Code, so the notebook format you have already authored against is unchanged.
  • The skill. The pedagogy — diagnose before answering, prefer questions to scaffolds and scaffolds to snippets, escalate, never reproduce the solution — lives in SKILL.md in the lab repo. It follows the Agent Skills spec, so the same file works in Claude Code and other compliant assistants.
  • Tool restriction. The Tutor agent’s tools: list omits editing and code execution. In v0.1, “don’t write the solution into their file” was a request to the model. Now it is a capability the tutor doesn’t have.

Setting it up for your course

Step 1 — Prepare your solutions repo

A private repo with one Quarto file per lesson. The structure, which templates/solution-template.qmd demonstrates:

# Sum the even numbers in a vector `{r-lab-1}`

**To-do:** Write a function `sum_even(x)` that returns the sum of all even numbers.

::: {.callout-caution collapse="true" title="Solution"}

```r
sum_even <- function(x) sum(x[x %% 2 == 0])
```

**Key points:**

- `x %% 2 == 0` produces a logical vector marking even entries.
:::

Two requirements, one of which the v0.1 docs got wrong:

  • The heading must contain the task ID inside bare braces: {r-lab-1}. The parser matches the literal {r-lab-1} including the braces. The old tutorial claimed a Quarto anchor like {#sec-r-lab-1 .task} would also work because the anchor contains the bare token — it does not, and it never did. The shipped template always used the correct form, so notebooks written from it are fine.
  • The solution must be wrapped in a callout titled "Solution". Any callout type works; only the title matters.

Step 2 — Deploy

Railway project → add Postgres → add a service from this repo with Root Directory server. Set DATABASE_URL to the Postgres reference, and DASHBOARD_PASSWORD to a long random string if you want the hosted dashboard (any username, that password). Then from server/:

npm install
npm run migrate                                  # create the tables
npm run add-course -- my-course-2026             # issue the class token — printed once
npm run load -- ../../my-solutions --course my-course-2026

The token is the course identity, so one server and one database serve any number of classes, and a token issued for one cannot reach another’s solutions. The loader refuses a course that does not exist, which makes the mistake hard to make.

curl https://<app>.up.railway.app/healthz should return {"ok":true}.

Then check that every exercise students can ask about actually has a solution behind it — the one failure a clean load cannot catch, because it spans two repos:

npm run verify -- ../../my-lab-repo/labs --course my-course-2026

It exits non-zero and names the task IDs, so you can gate a publish on it.

Step 3 — Wire up the lab repo

Copy templates/lab-repo/.posit/ into the repo your students clone, and templates/lab-repo/README.md too, filling in the disclosure section.

That gives students the skill and the Tutor agent. It deliberately contains no settings.json, and this is the one thing worth knowing before you spend an evening debugging it.

A workspace .posit/assistant/settings.json does configure mcpServers, and it takes precedence over the user-level file the installer writes. So shipping one in the lab repo silently overrides every student’s working install. If it uses an {env:...} placeholder for the token or username, nothing expands it — the literal string is sent as a header, the server rejects it, and Posit Assistant drops the server. What the student sees is a tutor that loads, tutors, and has no tools, with nothing anywhere pointing at the cause. Skills and agents from a workspace are fine; only the server config has to come from the installer.

Step 4 — Point students at the installer

Change the url default in install.R to your server, then give students the class token. They run one line in the R console:

source("https://raw.githubusercontent.com/benrosche/socratic-tutor/master/install.R")
install_tutor(token = "...", student = "their-github-username")

It writes user-level config, then calls your server and reports the course, the identity the server sees, and how many exercises are loaded — so a student knows immediately whether it worked, rather than discovering it mid-lab. tutor_check() re-runs that test later; uninstall_tutor() reverses it.

One thing you get for free by copying the template: its settings.json carries a permission block pre-approving the two tutor tools. Without it, the first lookup raises a permission prompt, and a student who dismisses it gets a tutor that keeps talking but can no longer see the reference solution — hints that are suddenly generic, with no error to explain why. Thirty students, thirty chances to click the wrong button, and the failure looks like your tutor is bad rather than un-permitted. That file is the only settings.json a lab repo should ever contain.

The #| task: markers are already in the worksheets you generated. Just tell them not to delete them.

No extension to install, and the class token never lives in the lab repo.

Using it in another IDE

Everything above assumes Positron, because that is the integration I built and use. The server underneath is not tied to it. It speaks MCP over Streamable HTTP at POST /mcp, and it authenticates with two ordinary headers — Authorization: Bearer <class token> and X-Tutor-Student: <username>. Any client that can reach an HTTP MCP server and set headers can use it. It is also stateless: the escalation level lives in Postgres, not in the conversation, so nothing about the ladder depends on the editor.

In VS Code, that is a .vscode/mcp.json:

{
  "servers": {
    "tutor": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer YOUR-CLASS-TOKEN",
        "X-Tutor-Student": "your-github-username"
      }
    }
  }
}

In Claude Code, one command:

claude mcp add --transport http tutor https://your-server.up.railway.app/mcp \
  --header "Authorization: Bearer YOUR-CLASS-TOKEN" \
  --header "X-Tutor-Student: your-github-username"

The pedagogy travels too. SKILL.md follows the Agent Skills spec, so it works unmodified in Claude Code; in VS Code the natural home for it is a custom chat mode.

Three things need a real substitute, and one of them matters:

  • Tool restriction — the one that matters. What stops the tutor writing the answer into a student’s file is not the prompt, it is that the Tutor agent has no edit or execute tool. Reproducing that means finding the host’s equivalent: a subagent with a restricted tools: list in Claude Code, a custom chat mode with a tools list in VS Code. Where a client has no way to constrain tools per mode, the guarantee weakens back into a request, which is what v0.1 was.
  • The installer. install.R writes Posit Assistant’s settings.json and reads .Renviron. Elsewhere, students configure by hand with the snippets above, or you write the equivalent for your stack.
  • Editor context. The skill finds the #| task: marker in whatever the assistant attaches as context. If your client does not pass the active file, the student has to say the task ID — or the tutor needs a file-read tool, which is a capability you were otherwise happy for it not to have.

I have not run a full course through a non-Positron client, so treat the above as a map rather than a trip report. The server side I am confident about; it is the same MCP endpoint either way.

Customizing the tutor for your domain

The default skill is deliberately generic — it says “programming exercises” and avoids naming a language. For your course you will want to name the language, mention the libraries students should reach for first, and swap the Socratic example questions for ones in your domain’s vocabulary.

Try it

The code is at github.com/benrosche/socratic-tutor. If you adopt it for your course, I’d be interested to hear how it goes — both the wins and the places where the model’s hint quality breaks down.