Run simulations with a local coding harness

Connect an installed coding harness as an external simulation target. Runtype runs the simulated users and grading. Your machine runs the harness and its configured Model Context Protocol (MCP) tools.

The CLI manages the local host, device listener, external Agent, and simulation target. You do not need a repository checkout or a separate wrapper process.

Configuration	Exact versions	Verification scope
Claude Code, native backend	Claude Code `2.1.314`, Agent SDK `0.3.308`	Installed runtime and local simulation report with a fixture provider
Codex, native backend	Codex `0.154.0`, app-server protocol	Installed runtime and packed CLI simulation report with a fixture provider.
Claude Code, Agent Client Protocol (ACP) backend	Claude Code `2.1.314`, `claude-agent-acp` `0.99.1`, Agent SDK `0.3.312`, ACP `1`	Installed adapter and packed CLI simulation report with a fixture provider.

A fresh Codex report also verifies per-turn provenance, the report-wide capture summary, and the declared profile and request-limit qualifications.

Installed-runtime checks use local fixtures on macOS. Linux process supervision has separate tests. These checks do not establish staging deployment or live provider compatibility. The host supports macOS and Linux and rejects Windows. Other version combinations are rejected. Codex through ACP and the workspace profile are unavailable.

Prepare the connection

Use Node.js 22.12.0 or later and pnpm.

Install a CLI release that includes runtype harness:
Install the Runtype CLI
```
pnpm add --global @runtypelabs/cli
```
Sign in to the Runtype organization that owns the simulation:
Sign in to Runtype
```
runtype auth login
```
For an API key, grant AGENTS:READ, AGENTS:WRITE, DEVICES:READ, DEVICES:WRITE, DEVICES:EXECUTE, SIMULATIONS:READ, SIMULATIONS:WRITE, SIMULATIONS:RUN, and TELEMETRY:WRITE. Use a production-mode API key, including against a staging API deployment. Test keys cannot authorize device execution or simulation runs, even when the requested permissions include those scopes.

Use an API deployment that supports signed harness telemetry. The host validates the per-turn binding and telemetry permission before starting a model turn. An older API deployment cannot run this host.
Install the exact harness version from the compatibility table. Authenticate with that harness before connecting. For Codex, install the pinned package:
Install Codex
```
pnpm add --global @openai/codex@0.154.0
codex login
```
For native Codex, the host copies file-backed auth.json and .credentials.json into an isolated configuration directory. It can also use OPENAI_API_KEY or CODEX_API_KEY. It does not copy your settings or previous sessions.

Keep the matching codex-code-mode-host binary installed with Codex. The controlled profile uses Codex Code Mode to call the configured MCP tools. It excludes built-in tool namespaces and disables the in-process fallback. A missing or disabled host fails readiness checks.

For Claude Code, use its installer to select 2.1.314 and sign in, or set ANTHROPIC_API_KEY. Use --executable /absolute/path/to/claude or --executable /absolute/path/to/codex to select an installed binary. The Runtype CLI does not silently install or upgrade the harness.
Create mcp.json with only the servers the simulation may call:
mcp.json
```
{
  "mcpServers": {
    "fixture": {
      "url": "http://127.0.0.1:4555/mcp"
    }
  }
}
```
Remote URLs require HTTPS. HTTP is allowed for loopback servers. Each server can include a headers object for authentication. Keep files that contain credentials private. Start the fixture server before checking the harness.

External tool writes are real. Use read-only tasks or a server where writes are safe. The controlled profile limits available tools, but it does not turn a write tool into a simulation. External targets report isolation as unverifiable.

Check and connect the harness

Check the installed runtime, authentication, and MCP discovery without starting a model turn:
Check Claude Code
```
runtype harness doctor --harness claude --backend native --mcp-config ./mcp.json
```
For Codex, replace --harness claude with --harness codex. A successful check prints ready: true. This confirms local credential detection and MCP discovery. It does not verify credential expiry, access to the requested model, or the model’s ability to call tools.
Start the connection:
Connect Claude Code
```
runtype harness connect \
  --harness claude \
  --backend native \
  --profile controlled-mcp \
  --mcp-config ./mcp.json \
  --data-policy redacted \
  --name "Read-only MCP harness"
```
Keep this command running. It prints the local registration ID and the paired device, Agent, and target IDs. Save the registration ID for status and cleanup commands.

Use --model MODEL_ID to request a model supported by that installed harness and account. Omitting the option uses the harness’s default model. To limit each session, set --max-turns from 1 to 100, with a default of 20. Set --timeout-ms from 1000 to 300000 to limit a turn, with a default of 300000. The host selects a free loopback port unless you set --port.
Use the printed target ID in a simulation variant. Start with one trial and read-only tasks that require the configured MCP tools. Set a low simulation spend limit for the Runtype work.
Open the report after the run completes. Inspect Harness capture for each turn and Open trace for its execution. Verify successful tasks, expected tool calls, and captured results. A completed run or accepted telemetry alone does not prove that the harness used a tool. Unknown outcomes remain unknown.

Use Claude Code through ACP

Install the pinned adapter into a directory you manage outside a project workspace. The adapter is a separate process that translates ACP requests into Claude Agent SDK calls.

Install the pinned ACP adapter

mkdir -p "$HOME/runtype-adapters/claude-acp"
pnpm --dir "$HOME/runtype-adapters/claude-acp" add --save-exact @agentclientprotocol/claude-agent-acp@0.99.1

Set ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, or CLAUDE_CODE_OAUTH_TOKEN in the local environment. This backend uses an isolated configuration directory. It does not read a local Claude login file or execute apiKeyHelper.

Check the ACP connection

runtype harness doctor \
  --harness claude \
  --backend acp \
  --adapter "$HOME/runtype-adapters/claude-acp/node_modules/.bin/claude-agent-acp" \
  --executable /absolute/path/to/claude \
  --mcp-config ./mcp.json

Use the same options with runtype harness connect. The backend denies permission requests outside the controlled MCP profile. The SDK aggregate can include adapter helper requests, such as title generation, without exposing each request. ACP usage coverage remains partial because those requests are not itemized.

Choose the capture policy

The default --data-policy redacted retains allowed content after secret and personal-data redaction. metrics omits message, tool-input, tool-output, and native-event content. full retains allowed content while still scrubbing known secrets.

The stricter host or organization policy wins. A host configured for full cannot override an organization policy that permits only metrics. Coverage in the report describes the retained evidence, so missing content is not presented as complete capture.

Tool evidence includes stable call identity, names, available input and output, failure details, and known timing. Missing outcomes or durations remain unknown. Coverage distinguishes available, unavailable, disabled, partial, redacted, and truncated fields. Delivery accepted confirms that submitted batches were accepted. It does not mean that the backend exposed every field.

Native observations are bounded to 128 KiB per event and 2 MiB per turn. Upload batches are bounded to 512 KiB. The local retry spool is bounded to 16 MiB and expires files older than 24 hours when initialized. Upload retries and final delivery waits are bounded. A failed upload can leave partial evidence and retained spool files. Restarting does not guarantee replay of an expired turn binding.

Reports include delivered, observed, and dropped event counts and unresolved tools. The report-wide capture summary keeps at most 200 turns and 256 KiB of turn summaries. It states how many summaries were omitted. Inspect individual conversations for per-turn evidence.

Interpret harness costs

Cost per success can display a Harness estimate when platform-measured cost is unavailable. This requires complete reported cost and at least one successful conversation. Partial reporting shows a known subtotal and coverage instead. Missing usage remains unknown.

The estimate describes the harness’s reported scope and pricing basis. It is not a subscription charge or a provider invoice. Reported harness costs do not affect platform billing or simulation spend caps. A simulation spend cap does not cap external provider charges or subscription use. CSV and Markdown exports preserve the estimate label and completeness.

Check status and clean up

Inspect the registration from the same API, organization, and user:

Inspect the connection

runtype harness status YOUR_REGISTRATION_ID --json

Status includes whether the local host is online and its telemetry delivery counters. To stop the host and listener while keeping the registration:

Disconnect the harness

runtype harness disconnect YOUR_REGISTRATION_ID

The command waits for local teardown and registration-lock release. Successful shutdown reports phase: stopped. If cleanup fails, inspect the reported stage before reconnecting or removing the registration.

Resume the stored configuration with:

Resume the harness

runtype harness connect --registration YOUR_REGISTRATION_ID

Resuming restores the registration, not an in-flight provider conversation. Start a new simulation run after an interruption.

After disconnecting, remove the registered resources with:

Remove the registration

runtype harness remove YOUR_REGISTRATION_ID --yes

Removal deletes the owned target and Agent and revokes the paired device. The private local registration record and retained telemetry files remain under the Runtype configuration directory. Remove those local files separately after you no longer need diagnostics.

Troubleshoot a connection

If doctor rejects a version, install the exact compatible version and pass its executable path. Do not bypass the version check.
If pairing or a run returns a permission error, check the key’s effective scopes and environment. A test key is insufficient for this workflow.
If a turn refuses its telemetry binding, check that the API supports harness telemetry and that the key, device, Agent, and simulation belong to the same authorized organization and device owner.
If the device disconnects, later turns fail with a device-unavailable error. Review partial evidence before reconnecting and starting a new run.
If the Agent’s protocol, transport, or device changes during a run, the run refuses later turns. Restore the registration or create a new one before rerunning.

Next steps

Bring your own agent: connect other external Agents.
Report telemetry from an external agent: inspect the telemetry contract.
Design a tool surface with simulations: define tasks and compare variants.