Experiments

AgentV eval files are the runnable authoring artifact. Use top-level description for display metadata, tags.experiment as the run/result grouping label, providers for the system under test, and flat top-level run controls such as timeout_seconds and threshold. Use evaluate_options for evaluation runtime options such as repeat, budget_usd, and max_concurrency. agentv eval --workers N remains an operator-side override.

name: support-regression
description: Support regression suite
tags:
  experiment: support-codex
providers:
  - id: codex-app-server
    label: support-codex
    extends: codex-gpt5
    config:
      model: gpt-5.1
      reasoning_effort: high
timeout_seconds: 720
evaluate_options:
  repeat: 4
  budget_usd: 2.00
  max_concurrency: 3

environment:
  type: host
  workdir: ./workspaces/support-codex
  setup:
    command: ["bash", "-lc", "bun install && bun run build"]
    cwd: "."
prompts:
  - "{{ input }}"

tests:
  - id: refund-eligibility
    vars:
      input: Can this customer get a refund?
    criteria: Applies the refund policy correctly

Layout Conventions

Use directories for human organization, not schema behavior. A common layout is:

evals/
  suites/
    refunds.eval.yaml
  cases/
    refund-smoke.cases.yaml
experiments/
  refunds-codex.eval.yaml

In that layout, evals/suites/refunds.eval.yaml is a reusable task suite, evals/cases/refund-smoke.cases.yaml is raw case data, and experiments/refunds-codex.eval.yaml is a normal eval file with its own run controls:

experiments/refunds-codex.eval.yaml

name: refunds-codex
providers:
  - codex-gpt5
prompts:
  - "{{ input }}"

tests:
  - file://../evals/cases/refund-smoke.cases.yaml
  - id: local-edge-case
    vars:
      input: Check a damaged final-sale refund.

The experiments/ folder is optional and user-owned. AgentV does not scan it for special files or infer runtime behavior from the path; the same eval could live under evals/, benchmarks/, or beside the suite it runs.

Multiple Suites And Raw Case Files

Use direct eval files for suites that own task context. Reference reusable raw case files from the local tests field when those rows should run in the current suite context.

tests:
  - file://cases/*.cases.yaml
  - file://cases/regression.jsonl
  - file://cases/smoke/*.cases.yaml

String-valued tests and string entries inside tests[] are raw-case import shorthand and may point at raw case files, directories, or globs. They do not load another eval suite’s runtime blocks. Run multiple eval suites directly with the CLI and use tags such as tags.experiment or domain-specific tags for grouping and filtering.

Scoped Run Overrides

Use test-level run: blocks for result interpretation and scheduling policies that vary by case. Precedence is:

test.run > parent top-level run controls

providers:
  - agent
threshold: 0.8
evaluate_options:
  repeat: 3
timeout_seconds: 300
tags:
  area: agentic
prompts:
  - "{{ input }}"

tests:
  - id: critical-case
    vars:
      input: "..."
    criteria: Must pass exactly
    run:
      threshold: 1.0
      budget_usd: 0.50

Scoped run: supports threshold, repeat, timeout_seconds, and per-case budget_usd overrides. Parent suite budgets should use evaluate_options.budget_usd for public eval authoring. Use evaluate_options.max_concurrency for authored concurrency. Candidate-changing fields stay parent-level. Coding-agent testbed setup belongs in environment, lifecycle hooks belong in top-level extensions, and provider-specific setup belongs in target configuration.

Lifecycle Ownership

Run controls do not own commands that prepare files, dependencies, repos, or target-specific runner state.

Need	Put it in
Install dependencies, build the repo, seed files	extensions: ["file://scripts/setup.mjs:beforeAll"]
Apply per-case state	extensions: ["file://scripts/setup.mjs:beforeEach"]
Reset file state after each case	environment reset policy
Configure an agent runner or provider variant	providers entry or providers.yaml
Choose the provider	top-level providers or CLI --provider
Override the provider’s default model	providers[].config.model
Configure repeat sample count, budget, concurrency, timeout, threshold	evaluate_options.repeat, evaluate_options.budget_usd, evaluate_options.max_concurrency, timeout_seconds, threshold
Bind an existing local workspace directory	--workspace-path or .agentv/config.local.yaml

extensions:
  - file://scripts/build.mjs:beforeAll

providers:
  - id: codex-app-server
    label: codex-with-skills
    extends: codex-gpt5
    hooks:
      before_each:
        command: ["sh", "-c", "cp -R skills \"{{workspace_path}}/.codex/skills\""]
evaluate_options:
  repeat: 3

Existing local workspace paths are machine-local bindings: pass --workspace-path for a one-off run or put execution.workspace_path in .agentv/config.local.yaml. Put coding-agent testbeds, workdirs, Docker config, repository setup, services, and reset policy under top-level or case-level environment; put lifecycle hooks in extensions and provider environment overrides in env.

Repeat Runs

Use evaluate_options.repeat when you want AgentV to try each case more than once:

evaluate_options:
  repeat: 3

Repeat authoring only sets the sample count. A fatal post-run aggregate policy surface is future work; until that exists, parse the run bundle externally when CI needs aggregate gating across samples. Per-case tests[].options.repeat overrides the global repeat count for that case.

Result Layout

Eval runs write to a direct run bundle:

.agentv/results/<run_id>/

CLI --experiment sets the experiment label explicitly. Without that flag, AgentV uses the reserved tags.experiment key (see below), then the suite name, then the eval filename. The precedence is --experiment > tags.experiment > default. There is no top-level experiment field — a run is labeled with tags.experiment. The Dashboard uses “Experiment” for the comparison and result grouping concept; folder names are only storage allocation and must not define result semantics.

Tags as run metadata (tags.experiment)

Suite-level tags accepts either the existing selection form (a string or list of strings that drives select.tags / --tag name filtering) or a metadata map:

tags:
  experiment: baseline-v2
  team: compliance

The map form is run metadata, not selection. The reserved experiment key feeds the experiment namespace, and the full map is emitted to summary.json.metadata.tags and every index.jsonl row so the Dashboard can group trend/compare views by tags.experiment.

Set or override map tags from the CLI with a repeatable --tag key=value flag (--tag experiment=baseline-v2 --tag team=compliance); bare --tag name keeps its existing file-selection meaning. Tags merge with precedence CLI --tag key=value > project config tags > eval tags. --experiment still wins over tags.experiment for the namespace, and an explicit --tag experiment= clears the label back to the default.

Imported source suite metadata appears in index.jsonl rows and manifests. Use index.jsonl fields such as eval_path, test_id, target, and result_dir for identity and artifact discovery instead of reconstructing paths from suite names or wrapper layout.

For the complete result file contract, including why row metadata is semantic truth and directories are storage allocation, see Result Artifact Contract.