How to test a new AI coding model on your own project before you switch

A new model shows up in your model picker almost every month, and each one arrives with a launch post full of impressive numbers. Those numbers are real, but they measure the vendor's tasks, not yours. Google's Gemini 4 Argon launch, for example, leans heavily on results from Google's own codebases.

The only way to know whether a model is better for you is to test it on your own work. That sounds like a research project, but a useful version takes an afternoon. In AI terms, this is called an eval: a fixed set of tasks plus a way to grade the results.

Step 1: Collect 10-20 real tasks

Don't invent tasks. Pull them from your actual history:

• Bugs you fixed in the last month (you know the right answer)
• Small features you already shipped
• Places where your current AI assistant got it wrong or got stuck

Anthropic's engineering team, in Demystifying evals for AI agents, recommends exactly this: "20-50 simple tasks drawn from real failures is a great start." For a solo builder comparing two models, 10-20 is enough to spot a clear difference.

Make each task unambiguous. Anthropic's test is that a good task is one where two experts would independently agree on pass or fail. "Make the dashboard nicer" fails that test. "The /api/orders endpoint returns 500 when status is missing; make it return 400 with an error message" passes.

Step 2: Write down how you'll grade each task

Decide the grading rule before you run anything, or you will grade generously for the model you already like. There are three kinds of graders:

• Code-based: a test passes, the build succeeds, a lint rule is clean. Fast, cheap and objective. Use these wherever you can.
• Model-based: another AI scores the output against a rubric. Flexible, but less consistent, so check it against your own judgment.
• Human: you read the result. The most accurate and the slowest.

Grade the outcome, not the path. If a model fixes the bug in a different way than you did, that still counts, as long as the tests pass and the code is something you'd merge.

A simple task file can look like this:

## Task 07 — missing status returns 500
Branch: eval/task-07 (reset to commit before the fix)
Prompt: "POST /api/orders crashes with 500 when `status` is omitted.
Return 400 with {error: 'status is required'} instead. Add a test."
Pass if: `npm test -- orders` passes AND the new test fails on the old code.

Step 3: Run each model the same way

Keep everything identical except the model: same starting commit, same prompt, same tools and permissions. Reset the branch between runs so one attempt doesn't leak into the next.

Switching models is quick in most tools:

• Claude Code: start a session with claude --model opus, or switch inside a session with /model sonnet. Per the model configuration docs, you can also pin an exact version by its full name (for example claude-opus-5-5), so your results don't silently change when an alias moves to a newer model.
• GitHub Copilot Chat: use the model dropdown at the bottom of the chat view in VS Code, Visual Studio, JetBrains, Eclipse or Xcode, as described in GitHub's docs. Note that changing the chat model doesn't change inline suggestions, which have their own setting.

Step 4: Run each task more than once

AI models are not deterministic. The same prompt can succeed on one run and fail on the next. Anthropic's post explains two ways to score repeated runs:

• pass@k: the chance that at least one of k attempts succeeds. Useful when you'll retry until it works.
• pass^k: the chance that all k attempts succeed. Useful when you need consistency.

The gap between them is bigger than it looks. Anthropic's example: a model that succeeds 75% of the time per attempt has only about a 42% chance of succeeding three times in a row. If you're handing a model long, unattended tasks, consistency is what you're actually buying.

Three runs per task per model is a reasonable starting point.

Step 5: Track cost and time, not just pass rate

For every run, record:

Task	Model	Passed?	Time	Cost / tokens
07	Model A	3/3	4 min	...
07	Model B	2/3	2 min	...

A model that passes slightly more tasks but costs three times as much may not be worth it for routine work. Also check whether the price you're seeing is permanent. Gemini 4 Argon's API price, for example, doubles after its introductory period ends on November 30, 2026.

Step 6: Read the transcripts

Don't just read the scoreboard. Anthropic's advice is blunt: "You won't know if your graders are working well unless you read the transcripts." Reading what each model actually did tells you whether a failure was the model's mistake or your grader rejecting a valid solution. It also shows you things a pass rate can't, like a model that passes the test but quietly deletes a check, or one that asks good clarifying questions.

Keep the eval and reuse it

Once you've built the task set, keep it in your repo. The next time a new model launches, rerunning your eval takes minutes, not an afternoon. Over time, tasks the models always pass become a regression check (does the new model still handle the basics?), and the tasks they keep failing become your list of what to watch for in the next release.

That's how you choose a model: not from the launch post, but from evidence on your own code.