autocontext

Name: autocontext
Author: Grey Haven

Agent evaluation and self-improvement harness exposed to MCP clients as scenarios, runs and judges

By Grey Haven TypeScript Local · stdio AI and ML Testing Developer tools

GitHub npm Website

43 popularity score ★ 1.3k ↓ 209/wk

Connect your client

Local (npm)

Add in one click

Add to Cursor Add to VS Code Add to LM Studio Add to Goose

Opens the app, which asks before it adds the server. Then replace each <NAME> in its settings with your value.

Or add it by command or config

It needs ANTHROPIC_API_KEY. Replace each <NAME> with your value.

Claude CodeTerminal

claude mcp add autocontext -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> -- npx -y autoctx serve mcp

Claude Desktopclaude_desktop_config.json

{
  "mcpServers": {
    "autocontext": {
      "command": "npx",
      "args": [
        "-y",
        "autoctx",
        "serve",
        "mcp"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
      }
    }
  }
}

Cursor~/.cursor/mcp.json

{
  "mcpServers": {
    "autocontext": {
      "command": "npx",
      "args": [
        "-y",
        "autoctx",
        "serve",
        "mcp"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
      }
    }
  }
}

VS Code.vscode/mcp.json

{
  "servers": {
    "autocontext": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "autoctx",
        "serve",
        "mcp"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<ANTHROPIC_API_KEY>"
      }
    }
  }
}

Codex~/.codex/config.toml

[mcp_servers.autocontext]
command = "npx"
args = ["-y", "autoctx", "serve", "mcp"]

[mcp_servers.autocontext.env]
ANTHROPIC_API_KEY = "<ANTHROPIC_API_KEY>"

Gemini CLITerminal

gemini mcp add --scope user -e ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY> autocontext npx -- -y autoctx serve mcp

Settings 7

`ANTHROPIC_API_KEY`	Required: Key for the default anthropic provider; the server exits at start without it. Not needed when another provider is chosen; AUTOCONTEXT_AGENT_API_KEY also works.
`AUTOCONTEXT_AGENT_PROVIDER`	Optional: Model provider: anthropic (default), openai, openai-compatible, gemini, mistral, groq, openrouter, azure-openai, ollama, vllm, hermes, claude-cli, codex, pi, pi-rpc or deterministic.
`AUTOCONTEXT_AGENT_API_KEY`	Optional: Generic API key used for whichever provider is selected.
`AUTOCONTEXT_AGENT_BASE_URL`	Optional: Endpoint for openai-compatible or self-hosted providers.
`AUTOCONTEXT_AGENT_DEFAULT_MODEL`	Optional: Model name to use instead of the provider's default.
`OPENAI_API_KEY`	Optional: Key read for the openai and openai-compatible providers when no generic key is set.
`AUTOCONTEXT_PI_COMMAND`	Optional: Command used to start Pi when the provider is pi.

What it does

autocontext is a harness for making agents better at a task over repeated attempts. You give it a goal or a saved scenario; it has a model attempt the task, scores the attempts with a judge or in matches, keeps what worked as playbooks, hints and skill notes, and records dead ends so later runs skip them. Everything lands on disk as run folders, traces, reports and a SQLite database.

The npm package autoctx is mainly a CLI; `autoctx serve mcp` starts its MCP server over stdio. Through it a client can judge an output against a rubric, run improvement loops, start and watch scenario runs and tournaments, submit background solve jobs, read the knowledge earlier runs stored, run verifier-checked missions and campaigns, record human feedback, and manage a local store of production traces that can be turned into evaluation datasets.

The tool surface is much larger than the README suggests. The README says "40+ tools"; the server code at the ts-v0.19.0 tag registers 78 tool names: 75 tools plus `autocontext_solve_scenario`, `autocontext_solve_status` and `autocontext_solve_result`, which are snake_case-argument aliases of the solve tools (ts/src/mcp/server.ts and the files it imports). The server also announces itself to clients as version 0.2.3, a fixed string in server.ts, whatever the installed package version is.

What it needs

Runtime	Node.js 22+

Node.js 22.19.0 or newer (npx runs the autoctx package). A model provider: the server picks `anthropic` unless AUTOCONTEXT_AGENT_PROVIDER says otherwise, and exits at start without ANTHROPIC_API_KEY (or AUTOCONTEXT_AGENT_API_KEY). Other providers need their own key or a local endpoint (ollama, vllm); `deterministic` needs none but does no real model work. The claude-cli, codex and pi providers need those programs installed. Model calls cost money with paid providers.

Limits

Its own integration guide says users have had better results driving autocontext through the CLI than through MCP. The SQLite database, runs and knowledge folders default to runs/autocontext.sqlite3, runs and knowledge relative to the directory the client starts the server in (ts/src/config/app-settings-schema.ts), so different clients may write to different places. Some workflows (ecosystem, ab-test, resume, wait, trigger-distillation) and real model training exist only in the Python package. The instrument tool's own description says its detector plugin registry may ship empty, so a scan can find nothing.

Who it suits

Suits developers who already run autocontext and want an assistant to start runs, judge outputs and read back what earlier runs learned. It is local, writes files in its working directory and calls whichever model provider you configure, so plan for API costs. Anyone who only wants to grade a few outputs, or who prefers predictable scripting, may be better served by the autoctx CLI directly, as its own docs recommend; the Python package is the pick for training and the Python-only workflows.

Tools 60 of 78

evaluate_output: Scores one output against a rubric in a single pass
run_improvement_loop: Runs several rounds of judging and revising an agent output
run_repl_session: Runs a REPL-style session to generate or revise an agent-task output
queue_task: Adds a task to the background runner queue
get_queue_status: Summarises the background task queue
get_task_result: Fetches the result of a queued task by id
capabilities: Reports version, scenarios, providers and features of this install
list_scenarios: Lists the available scenarios with their metadata
get_scenario: Shows the details of one scenario
revise_scenario: Rewrites a scenario spec from user feedback using the configured model
list_runs: Lists recent runs, with optional filters
get_run_status: Shows a run's progress, scores and generations
get_playbook: Reads the playbook built up for a scenario
run_scenario: Starts a scenario run with the given options
get_generation_detail: Shows the full results of one generation
list_runtime_sessions: Lists recent runtime-session event logs
get_runtime_session: Reads one runtime-session log by session or run id
get_runtime_session_timeline: Reads the operator timeline of a runtime session
validate_strategy: Checks a strategy JSON against a scenario's rules
run_match: Plays a single match of a scenario
run_tournament: Plays N matches and ranks strategies by Elo
read_trajectory: Returns a run's score history as markdown
read_hints: Reads the stored competitor hints for a scenario
read_analysis: Reads the analyst write-up for one generation
read_tools: Reads the tools the architect role generated for a scenario
read_skills: Reads the skill notes kept for a scenario
export_skill: Packages a scenario's knowledge as an installable skill with markdown
list_solved: Lists scenarios that have exported knowledge or finished runs
search_strategies: Searches earlier strategies by keyword
record_feedback: Stores human feedback on a scenario evaluation
get_feedback: Reads the human feedback stored for a scenario
run_replay: Reads the replay JSON of one generation
solve_scenario: Submits a plain-language problem as a background solve job
solve_status: Checks how a solve job is going
solve_result: Returns the skill package a finished solve job produced
sandbox_create: Creates an isolated sandbox for running a scenario
sandbox_run: Runs one or more generations inside a sandbox
sandbox_status: Shows a sandbox's state
sandbox_playbook: Reads the current playbook of a sandbox
sandbox_list: Lists the active sandboxes
sandbox_destroy: Removes a sandbox and deletes its data
create_agent_task: Saves a named agent-task spec for evaluation
list_agent_tasks: Lists the saved agent-task specs
get_agent_task: Reads one agent-task spec by name
generate_output: Produces a first agent output for a task prompt
export_package: Exports a versioned strategy package for a scenario
import_package: Merges a strategy package into a scenario's knowledge
create_mission: Creates a multi-step mission checked by verifiers
mission_status: Shows a mission's state and summary
mission_result: Returns a mission's full result with steps and checks
pause_mission: Pauses a running mission
cancel_mission: Cancels a mission
create_campaign: Creates a campaign that groups missions
campaign_progress: Shows how far a campaign has got
instrument: Scans a repo for LLM client calls and can patch in autocontext wrappers
production_traces_ingest: Validates incoming trace batches and moves them to the ingested store
production_traces_list: Lists locally stored production traces, unredacted
production_traces_build_dataset: Builds an evaluation dataset from curated traces
production_traces_export: Writes out traces with redaction applied
production_traces_prune: Deletes traces past the retention policy

Popularity

43 popularity score, on our list (the bar is 36)

★ 1.3k ↓ 209/wk

GitHub stars, whole repository	1,324	18.6 points
Stars in 30 days	+17	6.3 points
npm downloads, last week	209	18.6 points

It lives in the folder ts of greyhaven-ai/autocontext, which holds more than this server. Its stars are the whole repository's, so in the score they never count for more than its own downloads.

Measured 2026-11-08. How the score works.

Where it is published

Code	greyhaven-ai/autocontext/ts ★ 1.3k, the whole repository
npm	autoctx 0.19.1, 2026-10-26

What it can reach

Runs	On your machine
Reaches	Files on your machine, The internet, Your code
Changes things	Yes: it can write, send, create or delete
Sign-in	An API key or token
Licence	Apache-2.0 (npm)
Last code change	2026-11-05

Runs on your machine over stdio. It reads and writes a SQLite database plus run, knowledge and trace folders under its working directory, and sandbox_destroy and production_traces_prune delete data there. Most run, judge and solve tools send prompts and outputs to the configured model provider over the network using your API key. The instrument tool edits source files in a repo you point it at when mode is apply or apply-branch, and can create a branch and commit; it defaults to dry-run (ts/src/mcp/instrument-tools.ts). production_traces_list returns stored traces without redaction; redaction applies at export.

Over time

GitHub stars, whole repository 1.3k +17 (+1 %)

1.3k1.3k

13 Oct 20268 Nov 2026

npm downloads a week 209 −228 (−52 %)

84272

23 Mar 20268 Nov 2026

Before the site measured it, read from npm's daily download counts.

Popularity score 43 −12 (−22 %)

5630

13 Oct 20268 Nov 2026

What is rising across the directory

Sources

Page checked 2026-10-14.

Badge for your README

Shows this server's rank in its category, and stars for a server that runs on your machine. It updates every hour.

MarkdownREADME.md

[![autocontext on EveryMCPServer](https://everymcpserver.com/badge/autocontext.svg)](https://everymcpserver.com/servers/autocontext)

Connect your client

Add in one click

Or add it by command or config

Settings 7

What it does

What it needs

Limits

Who it suits

Tools 60 of 78

Similar servers

agent-harnesses MCP server

Agentic Flow

Prism Coder

AgentRQ

MartinLoop

mcpgrade