new AI models drop almost every week, and every launch post says the new one is the best yet.
if you run more than 1 AI, your real question is: does the new one get added to your stack or not?
don't decide from the launch post or the benchmarks. run your own tests. give the new model a trial on a task you already run, and let your own numbers find a slot for it in your stack.
pick 1 task you already run
start with something your AI does for you all the time: sorting posts, pulling quotes, summarizing a doc, tagging notes, grabbing analytics.
it has to be a task your AI already does and you have data on. that's what makes it a fair test, because you or your AI can check every output yourself.
then give the exact same task to your current model and the new one. same instructions, same input.
run the numbers
these are the 5 to look at:
- speed: how long it took
- accuracy: were the answers correct?
- token efficiency: how many tokens it burnt
- cost: what that run cost you
- compute: how heavy it is to run (this one matters most when a model runs on your own machine)
then put both outputs side by side and read them. a model can score well and still hand you something you wouldn't use.
[which one do you weigh the most: speed, cost or accuracy?]
what happened when 4 AI models took the test
the worked example: this week Auny had her Claude run the test on 1 of her common jobs: sort 12 posts into 6 categories, name the tool in each one, and quote the line that proves it. 2 runs per model, scored by code, 36 points a run.
- Haiku 5.5: 100%, 6.4 secs, 640 tokens in and 1,741 out, cost included through a Claude plan she'd pay for anyway
- Muse Wonder: 100%, 12.9 secs, about 35.7K in and 1.48K out, about $0.17 a run
- Qwen 3.7 Flash: 98.6%, 14.3 secs, about 77.7K in and 1.9K out, about $0.0026 a run
- GPT-5.6 Luna: 93.1%, 10.2 secs, about 24.1K in and 405 out, cost included through Codex on a ChatGPT plan
Haiku ended up being the best mix of accuracy, speed and cost for this kind of job.
1 easy task, 2 runs each: read a result like this as a direction, and keep testing.
have your agent do it
you don't need to build the test yourself. have your agent do this:
pick 1 task you run for me every week. run it on [current model] and [new model] with the same input, 2 times each. score each output against the right answer and give me a table: accuracy, time, tokens in and out, and cost per run. then show me both outputs side by side.
then you decide. if the new one wins, it gets the job. if not, BYE.
1:1 call · matches this post
AI Workflows for Creators
want help setting up AI for your creative process, tests and all? That's the AI Workflows for Creators call.
