Example Evaluations

This page collects complete eval file examples you can copy and adapt. Each demonstrates a different AgentV pattern.

Basic Q&A

A minimal eval with a single question and expected answer:

description: Basic arithmetic evaluation
providers:
  - default

prompts:
  - "{{ question }}"

tests:
  - id: simple-addition
    vars:
      question: What is 2 + 2?
      expected_output: "4"
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

Code Review with File References

Use multipart content to attach files alongside text prompts:

description: Code review with guidelines
providers:
  - azure-base

prompts:
  - - role: system
      content: You are an expert code reviewer.
    - role: user
      content:
        - type: text
          value: |-
            Review this function for security issues:

            ```python
            def get_user(user_id):
                query = f"SELECT * FROM users WHERE id = {user_id}"
                return db.execute(query)
            ```
        - type: file
          value: /prompts/security-guidelines.md

tests:
  - id: code-review-basic
    vars:
      expected_output: |-
        This code has a critical SQL injection vulnerability. The user_id is directly
        interpolated into the query string without sanitization.

        Recommended fix:
        ```python
        def get_user(user_id):
            query = "SELECT * FROM users WHERE id = ?"
            return db.execute(query, (user_id,))
        ```
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

Multi-Grader

Combine a script grader and an LLM grader on the same test:

description: JSON generation with validation
providers:
  - default

prompts:
  - "{{ input }}"

tests:
  - id: json-generation-with-validation
    vars:
      input: |-
        Generate a JSON object for a user with name "Alice",
        email "alice@example.com", and role "admin".
      expected_output: |-
        {
          "name": "Alice",
          "email": "alice@example.com",
          "role": "admin"
        }

    assert:
      - name: json_format_validator
        type: script
        command: [uv, run, validate_json.py]
        cwd: ./graders
      - name: content_evaluator
        type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"
        prompt: ./graders/semantic_correctness.md

File Output Transform

Convert a binary file output into text before the llm-rubric sees it:

description: Grade spreadsheet output via a transform

providers:
  - file_output
default_test:
  options:
    transform: file://../scripts/transforms/xlsx-to-csv.ts

tests:
  - id: spreadsheet-output
    vars:
      input: Generate the spreadsheet report
    criteria: The extracted spreadsheet content includes the revenue rows
    assert:
      - Output contains the transformed spreadsheet text including the revenue rows

See examples/features/file-transforms/ for a runnable end-to-end example with a file-producing target and custom grader target.

Trajectory Assertions

Validate that an agent uses specific tools during execution:

description: Tool usage validation
providers:
  - mock_agent

prompts:
  - "{{ input }}"

tests:
  # Validate minimum tool usage (order doesn't matter)
  - id: research-depth
    criteria: Agent researches thoroughly
    vars:
      input: Research REST vs GraphQL
    assert:
      - name: research-check
        type: trajectory:tool-used
        value:
          name: knowledgeSearch
          min: 2
      - type: trajectory:tool-used
        value:
          name: documentRetrieve
          min: 1

  # Validate exact tool sequence
  - id: auth-flow
    criteria: Agent follows auth sequence
    vars:
      input: Authenticate user
    assert:
      - name: auth-sequence
        type: trajectory:tool-sequence
        value:
          mode: exact
          steps:
            - checkCredentials
            - generateToken

Offline Grader Benchmark

Benchmark a five-model grader panel against a human-labeled export, then compare grader setups:

description: Offline grader benchmark
providers:
  - fixture_replay

tests:
  - file://../fixtures/labeled-grader-export.jsonl

assert:
  - name: grader-panel
    type: assert-set
    threshold: 0.6
    assert:
      - name: grader-gpt-5-mini
        type: llm-rubric
        provider: grader_gpt_5_mini
        prompt: ../prompts/grader-pass-fail-v1.md
      - name: grader-claude-haiku
        type: llm-rubric
        provider: grader_claude_haiku
        prompt: ../prompts/grader-pass-fail-v1.md
      - name: grader-gemini-flash
        type: llm-rubric
        provider: grader_gemini_flash
        prompt: ../prompts/grader-pass-fail-v1.md

See examples/showcase/offline-grader-benchmark/ for the full workflow, replay target, export contract, scoring script, and A/B compare commands.

Static Trace

Evaluate pre-existing trace files without running an agent:

description: Static trace evaluation
providers:
  - static_trace

prompts:
  - "{{ input }}"

tests:
  - id: validate-trace-file
    criteria: Trace contains required steps
    vars:
      input: Analyze trace
    assert:
      - name: trace-check
        type: trajectory:tool-sequence
        value:
          mode: in_order
          steps:
            - webSearch
            - readFile

Multi-Turn Conversation

Test multi-turn interactions where intermediate messages set context:

description: Multi-turn debugging session with clarifying questions
providers:
  - default

prompts:
  - "{{ input }}"

tests:
  - id: debug-with-clarification
    criteria: |-
      Assistant conducts a multi-turn debugging session, asking clarification
      questions when needed, correctly diagnosing the bug, and proposing a clear
      fix with rationale.
    vars:
      input:
        - role: system
          content: You are an expert debugging assistant who reasons step by step, asks clarifying questions, and explains fixes clearly.
        - role: user
          content: |-
            I'm getting an off-by-one error in this function, but I can't see why:

            ```python
            def get_items(items):
                result = []
                for i in range(len(items) - 1):
                    result.append(items[i])
                return result
            ```

            Sometimes the last element is missing. Can you help debug this?
        - role: assistant
          content: |-
            I can help debug this. Before I propose a fix, could you tell me:
            - What output you expect for an example input list
            - What output you actually get
        - role: user
          content: |-
            For `[1, 2, 3, 4]` I expect `[1, 2, 3, 4]`, but I get `[1, 2, 3]`.
      expected_output: |-
        You have an off-by-one error in your loop bounds.
        You're iterating with `range(len(items) - 1)`, which stops before the last index.
        To include all items, you can either:
        - Use `range(len(items))`, or
        - Iterate directly over the list: `for item in items:`

        Here's a corrected version:

        ```python
        def get_items(items):
            result = []
            for item in items:
                result.append(item)
            return result
        ```
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

Shared Prompt Context

Share a common prompt or system instruction across all tests with top-level prompts, then keep per-case data in tests[].vars:

description: Travel assistant evaluation
prompts:
  - - role: system
      content: |
        You are a knowledgeable travel assistant.
        Always include a practical safety tip.
    - role: user
      content: "{{ question }}"

tests: ./cases.yaml

cases.yaml

- id: japan-spring
  criteria: Recommends spring for cherry blossoms and mentions visa requirements
  vars:
    question: When is the best time to visit Japan?

- id: iceland-lights
  criteria: Recommends winter for Northern Lights
  vars:
    question: I want to see the Northern Lights in Iceland. When should I go?

- id: currency-only
  criteria: Provides direct answer about currency
  vars:
    question: What currency does Thailand use?

See the suite-level input migration example for a complete working version.

File Path Conventions

• Absolute paths (start with /): resolved from the repository root
  • Example: /prompts/guidelines.md resolves to <repo_root>/prompts/guidelines.md
• Relative paths (start with ./ or ../): resolved from the eval file directory
  • Example: ../../prompts/file.md goes two directories up, then into prompts/

Tips for Writing criteria

• Be specific about what success looks like
• Mention key elements that must be present
• For classification tasks, specify the expected category
• For reasoning tasks, describe the thought process expected

Tips for Writing expected_output

• Show the pattern, not rigid templates
• Allow for natural language variation
• Focus on semantic correctness over exact matching
• Graders handle the actual validation logic

Showcases

For complete end-to-end workflows that combine multiple features, see the showcases in examples/showcase/:

• Multi-Model Benchmark — weighted metrics × repeated runs × compare workflow. Run the same eval once per target, then use agentv results compare or Dashboard analytics to review the completed runs side-by-side.
• Export Screening — classification eval with confusion matrix metrics and CI gating.