Egil Hansen's .com

Verify snapshot tests and AI agents: disable the diff tool

I use Verify for snapshot testing in .NET projects. It is great, but there is one annoying edge case when an AI agent runs the test suite: a snapshot mismatch can make Verify ask DiffEngine to open my configured diff tool and wait for it to close.

For a local test run, that is exactly what I want. I run the test, the diff opens, and I can inspect the changed snapshot.

It is not what I want when an AI agent is working autonomously.

In that case the test process keeps running while it waits for the diff tool to close. The agent is waiting for the test command to finish. I am not looking at the screen. Nothing useful happens.

Even worse, if I am using the machine while the agent is working, the diff tool can suddenly pop up and take focus because a background test run found a snapshot mismatch.

The fix is a small guard that disables DiffEngine’s diff-tool launcher when the tests run under a known automated agent environment.

The trick is that AI coding agents usually set environment variables when they run commands. We can look for those and customize the behavior of our test suite, or other code that behaves differently in interactive and automated contexts.

The guard

Add this somewhere in the test project:

using DiffEngine;

/// <summary>
/// Disables the DiffEngine diff-tool launcher when running under an automated
/// agent or CI context, so a Verify snapshot mismatch never pops a UI window
/// that blocks the test run.
/// </summary>
/// <remarks>
/// <para>
/// Local interactive runs are unaffected: the diff tool still opens by default
/// when a snapshot mismatches, which is the desired developer experience.
/// </para>
/// <para>
/// Triggers (any one of the following sets <see cref="DiffRunner.Disabled"/> to
/// <c>true</c>):
/// <list type="bullet">
/// <item><description><c>COPILOT_CLI</c> / <c>COPILOT_AGENT_SESSION_ID</c> - GitHub Copilot CLI agent runs.</description></item>
/// <item><description><c>CLAUDECODE</c> / <c>CLAUDE_CODE</c> - Anthropic Claude Code CLI.</description></item>
/// <item><description><c>CODEX_SANDBOX</c> / <c>CODEX_CLI</c> - OpenAI Codex CLI.</description></item>
/// <item><description><c>AIDER</c> - Aider (set by aider when launching tools).</description></item>
/// <item><description><c>CURSOR_AGENT</c> - Cursor agent runs.</description></item>
/// <item><description><c>DiffEngine_Disabled</c> - DiffEngine's own opt-out.</description></item>
/// </list>
/// DiffEngine already self-disables in known CI environments
/// (<c>CI</c>, <c>TF_BUILD</c>, <c>GITHUB_ACTIONS</c>, etc.), so we do not
/// re-check them here.
/// </para>
/// </remarks>
public static class AgentDiffGuard
{
    private static readonly string[] AgentEnvVars =
    [
        "COPILOT_CLI", // GitHub Copilot CLI
        "COPILOT_AGENT_SESSION_ID", // GitHub Copilot CLI (session id)
        "CLAUDECODE", // Claude Code CLI
        "CLAUDE_CODE", // Claude Code CLI (alt)
        "CODEX_SANDBOX", // OpenAI Codex CLI
        "CODEX_CLI", // OpenAI Codex CLI (alt)
        "AIDER", // aider
        "CURSOR_AGENT", // Cursor
        "DiffEngine_Disabled" // explicit opt-out
    ];

    public static void Apply()
    {
        foreach (var name in AgentEnvVars)
        {
            if (HasEnv(name))
            {
                DiffRunner.Disabled = true;
                return;
            }
        }
    }

    private static bool HasEnv(string name)
        => !string.IsNullOrEmpty(Environment.GetEnvironmentVariable(name));
}

Then apply it once when the test assembly loads:

using System.Runtime.CompilerServices;

public static class TestModuleInitializer
{
    [ModuleInitializer]
    public static void Initialize()
    {
        AgentDiffGuard.Apply();
    }
}

That keeps local interactive runs unchanged, while making agent-driven test runs behave like normal command-line test runs: fail, print the error, and exit.

Install the Verify dotnet tool

I also recommend installing the Verify.Tool dotnet tool from the Verify.Terminal project:

dotnet tool install -g verify.tool

The tool makes it easier to deal with pending snapshots without opening a diff UI:

dotnet verify review
dotnet verify accept
dotnet verify reject

review shows the pending snapshots, accept updates the verified snapshots, and reject removes the received snapshots. There is also --work <DIRECTORY> if you need to point it at a specific working directory, and --yes for accept and reject when you want to skip the confirmation prompt.

That is useful for agents too. After a failing test run, the agent can inspect the diff in the terminal with dotnet verify review, decide whether the snapshot change is intentional, and then either fix the code or run dotnet verify accept --yes.

Why not just disable it everywhere?

Because the diff tool is still useful when I am the one running the test.

Snapshot tests are easiest to work with when the approval flow is quick. I want a diff to open locally when I intentionally changed rendered output or serialized data. I just do not want that UI interaction when the “developer” running the test is an automated agent that cannot use the diff tool anyway.

This guard keeps those two workflows separate:

Hope this helps.

Comments