How to compare AI image models fairly: a test grid you can repeat

Most comparisons of image generators go like this: one prompt, one render per model, a winner declared in thirty seconds. It feels like a test and it is closer to a lottery. One lucky roll on a weaker model beats one unlucky roll on a stronger one, and you walk away with a favourite chosen by chance. This post is a method instead: a small grid you can run in an afternoon on the studio's model catalog, with a sheet to write the results on.

The one rule: change one thing

A comparison is only fair if exactly one variable differs between the two renders you are judging. If the model and the prompt both changed, you have learned nothing about either. So decide first what the question is, and hold everything else still:

• Which model? Same prompt, same aspect ratio, same seed, same quality setting. Only the model changes.
• Which wording? Same model, same seed, same settings. Only one phrase changes. The seed post explains why pinning the number makes this possible.
• Which setting? Same model and prompt and seed, one option flipped.

Write the question at the top of your sheet before you press Generate. It sounds fussy, and it is the step that stops a test from drifting into three questions answered badly.

Build the grid

Pick two or three candidates from the image models on the catalog page, for example Still and Still Lite next to one or two others whose pages describe the job you have. Then pick three prompts that stress different things, not three versions of the same scene:

1. 01A scene with layout. A subject, a setting, light, and one specific placement ("the lamp on the left, the window behind"). This tests composition and how literally the model follows you.
2. 02A scene with text or hands. A shop sign with a short word, or a person holding a cup. These are the places where models differ most visibly. The hands and text post covers why.
3. 03A scene with material. Wet stone, brushed metal, wool, skin. This tests texture and lighting, the thing people call realism.

Write each prompt well before the test; a vague prompt rewards whichever model guesses your taste best, which is a different contest. The prompt guide has the structure. Then run every prompt on every candidate with the same seed and the same aspect ratio. Three prompts and three models is nine renders, which is a test you can afford.

What to look at, in this order

Judge each picture against the same short checklist, not against a general feeling. Give each item a score from one to three and write a word of evidence.

• Prompt adherence. Is everything you asked for there, in the place you asked? Count the missing items.
• Composition. Does the eye go where you wanted, is the framing usable, is there space where you need it?
• Text. Are the letters right, spelled right, on the right object?
• Hands and anatomy. Count the fingers. Check where limbs join.
• Texture and light. Zoom in. Does the material look like the material, are the shadows consistent with one light?
• Artefacts. Smeared edges, doubled objects, a pattern that repeats.

Judge before you look at which model made which picture, if you can. Ask a friend to rename the files, or cover the label. We all like the model we already believe in.

Log it like a lab notebook

A sheet with one row per render is enough: the date, the model name, the prompt label, the seed, the aspect ratio, any setting you changed, the six scores, and a line of notes. The library keeps your pictures and lets you reuse a prompt, but it does not bring back the advanced settings, so the seed lives on your sheet or nowhere. When you come back in a month, a row tells you exactly what a result was made from.

A gap or just noise?

Here is the uncomfortable part. Even with the seed pinned, a single render is one sample, and some of the gap between two pictures is luck. A few ways to tell the difference:

• Repeat the best and worst. Render the same prompt on the same model with two more seeds. If the "bad" model produces a winner on the second seed, the first gap was noise.
• Look for a pattern across prompts. One model missing the text in all three scenes is a finding. One miss in nine is a roll.
• Look at the kind of error. Consistent mistakes of one type (always blurred lettering, always a plastic sheen) come from the model. Random, different mistakes each time come from the dice.
• Mind the size of the gap. A tiny difference in score between two models is not worth acting on. A model that wins on eight of nine rows is.

The limits of the method

Say it plainly: a fair test is still a small sample, and nine renders say something about your three prompts, not about every picture you will ever make. Three more things can quietly break a comparison:

• Not every model takes a seed. Some keep it internal, and Fast Image has no seed field at all. For those you cannot pin the starting point, so you need more renders per cell, and a pattern across many matters more than any single picture.
• Pinned does not mean pixel-identical. Where a seed exists, fine detail can still vary a little between runs.
• Models change. A comparison is a photograph of the catalog on the day you ran it. If a model is updated, redo the rows that mattered.

Test cheaply, finish on the winner

The grid does not need to be expensive, because most of the learning is about your prompts and layouts, not about the finish. Start with the wording and composition on Fast Image, the cheapest model, where a bad idea costs little. Remember it is its own renderer, so a Fast Image draft is a sketch of the idea and not a preview of how another model will draw it. Once you have a prompt that works, run it on your two real candidates, pick the one that scored best on the items you care about, and spend your quality renders there only. The price shows on the Generate button before you commit, and the re-roll post shows how to count what a usable picture actually costs, which is the number that should settle a close call.

Video works the same way with higher stakes per render; the video model post applies this grid to clips. When you are ready to run yours, open the studio and start with the question line at the top of the sheet.