During the process of building our GenUI app for Google Cloud Next, the VGV Life Goal Simulator, I kept testing different LLMs. I wanted to know: which model was the fastest? Which models hallucinated GenUI catalog items or props, or failed to conform to the A2UI spec, something we call “structural hallucinations”? Which produced the best UIs?
At first, I swapped models out by hand to get a feel for how each one performed. I’d change the model, hot restart, and run through the flow. That’s a great way to build intuition, but it isn’t very scientific. I constantly opened the network tab, manually inspected the round-trip time, visually confirmed errors, and then asked colleagues “does this one feel faster?”
After running through this process a few times, I had an insight: Why don’t I just build a benchmark! So that’s what I did. I’d like to introduce you to the Very Good Ventures GenUI model benchmarks.
How should I build the benchmark? What should I measure?
At first, I started with a Flutter UI integration test. It ran the app, clicked through 3 rounds of back-and-forth with the real UI, and measured the average round-trip time for each request. I ran that 5 times against each model.
I wanted to measure more than simple round-trip times. I also wanted to know how many tokens each model used, how quickly the models responded with their first tokens, and how many responded with structural hallucinations.
Furthermore, I realized introducing a full-blown UI into the mix might make life a bit harder. The GenUI library hides some errors from end users, and Flutter UI integration tests are more complex to run on CI. Therefore, I decided to move from a Flutter UI integration test to a data-layer integration test talking directly to each model.
I think, therefore I am
My colleague Rémy Baudet built a GenUI prototype for a client, and I was blown away by how quickly his prototype returned results. I asked him to share his code, and I realized he had used an interesting technique: he disabled thinking mode! I hadn’t even thought of that.
After seeing his code, I knew I wanted to benchmark thinking vs non-thinking modes as well to understand the impact on round-trip time. Now, the benchmark measures supported models with thinking on and off.
Turning thinking off sounds simple, but every provider does it a little differently, and that took some fiddling to get right. Gemini lets you dial the thinking level down. OpenAI’s GPT-5 models have a reasoning effort setting, and the lowest it goes is “low”. DeepSeek needed a non-standard field in the request body that I had to inject by hand. And Kimi’s k2.7-code won’t let you turn thinking off at all, so it doesn’t get a no-thinking variant. If you try this yourself, budget a little time for the per-provider knobs.
One important note: The benchmark catalog doesn’t use any tools. If you put extra tools or agents in the pipeline, carefully test the results! That’s a gap I’d like to plug in the benchmarks.
Measuring round-trip
After implementing the test suite and initial models, my first run had two results that made no sense. First, GPT-5 mini was slower than full GPT-5. Second, a model with thinking turned off was slower than the same model with thinking on. I had manually tested these models and configurations, and I knew those results didn’t make sense.
I asked Claude Code to record round-trip information in log files so I could inspect the results. Looking into the logs, I realized the cause was the order of my tests, not the models.
At first, I ran all five passes of one model before starting the next. The result: each provider got hit in bursts and started rate-limiting! Whichever model ran later in the batch ate the throttling. The mini model wasn’t actually slower. It just ran after its bigger sibling.
To fix this, I changed the ordering of test execution. Now, the test suite runs one pass of each model, then reshuffles the order for the next round, so a provider’s requests spread out and no model gets blamed for running late. Finally, I also introduced a small delay between each test as an extra precaution to ensure we don’t measure rate limiting.
Average token count
As I looked into the numbers, I found two more issues: First, the total token count of all models was almost identical. Second, the time to first chunk varied widely by provider.
In the first version, I had combined input and output tokens together into “total tokens.” That’s not very helpful for understanding costs, as model providers charge differently for input and output tokens. Furthermore, total token counts came out nearly identical across models, because our system prompt and catalog are large and get sent every turn.
To fix this issue, I switched to counting output tokens only, as input tokens are nearly identical across all model tests given each model uses the same prompt and catalog.
Time to first chunk
I also wanted to measure how fast a user might see their first widget painted on screen. Therefore, I had the idea to measure “time to first chunk.” Basically, how fast a model starts returning its first tokens.
However, I discovered time to first chunk isn’t comparable across providers: Google sends a response in a handful of big chunks while OpenAI streams token by token, so OpenAI’s first chunk always shows up sooner even when Google may have a faster total round-trip.
The problem goes deeper. Getting those first tokens back quickly is a real advantage for plain text, where you can paint words on screen as they arrive. But a word of warning for GenUI experiences: time to first chunk does not mean “time to first catalog item shown to the user.”
Therefore, while it seemed like an interesting statistic to track when I started, our team found it more misleading than useful and decided to remove it from the benchmarks.
What counts as an “error”
Next, I wanted to measure an “error rate.” At first, a turn only failed if the request errored or produced nothing. Once again, I realized I was missing a big part of the picture.
The model builds each turn from our component catalog, and every catalog item has a JSON schema. The slider catalog item requires a value, a min, and a max. A radio card requires options. The model is supposed to match those, and doesn’t always. A slider might come back with no value, or a radio card with no options. In our runs, one Gemini model sent its component list as a bare array, instead of valid A2UI, on 6 of 15 turns.
When that happens, the GenUI library does the right thing for end users, but the wrong thing for a benchmark. The library notices the bad catalog item, logs it, asks the model to fix it, and renders the surface anyway. To a user, the turn looks like nothing bad happened. Sadly, it also meant my initial integration testing code recorded a success!
To catch those issues, I implemented additional checks. Now the benchmark validates the output against the catalog schemas, separately from the framework.
A turn fails if any component breaks its schema or uses a component that isn’t in the catalog. Furthermore, the benchmark checks for A2UI compliance. Those are real failures and should be captured as such.
False positives
There’s a flip side I had to watch, too: I needed to make sure my own error checks didn’t cry wolf. Going headless wasn’t entirely free here. Once I dropped the real UI, I no longer had a loading state to watch, so I had to decide when a turn was actually done.
My first version waited a fixed 300 milliseconds after each turn for the parsing to settle, then recorded the result. Sometimes a model returned a perfectly good surface a hair later than that, and my code logged a failure for a turn that had actually worked.
When I re-ran those models, a couple of the “failures” simply disappeared. To solve the issue, I swapped the fixed wait for one that watches until the results go quiet before running the error validation. A benchmark that reports failures that aren’t real is just as misleading as one that misses the real ones.
Benchmarking new models
One powerful implication of having a benchmark is that we can constantly update it with new models as they’re released. At the time of this writing, Gemini 3.6-flash and 3.5-flash-lite were just announced. How do they stack up against 3.5-flash and 3.1-flash-lite? Well, we’ve already plugged them into the GenUI model benchmarks, so you can view the results to see for yourself!
As new models are introduced, all we have to do is create a small PR, run the GitHub CI flow, and new results appear on the benchmark. Easy peasy.
It still boils down to taste
If you paid careful attention to the start of the article, you may have noticed I didn’t answer one key question: “Which model produced the best UIs?”
Benjamin Disraeli famously said there are three kinds of lies: lies, damned lies, and benchmarks. That’s a (bad) joke, of course, but the point stands! Benchmarks are great, but they only tell part of the story.
When it comes to GenUI, for certain use cases, you may want to slow down a bit. Perhaps you have tool calls or agent flows. Other times, you may want to display more complex dashboards or statistical information. If it’s more important to get those details right than to give the user something to look at as quickly as possible, try a slower model with higher levels of thinking enabled.
In other cases, perhaps you have all the information you need up front and want to show something as quickly as possible. In that case, respect your customers’ time and use a faster model with lower levels of thinking.
In the end, the benchmarks help us understand which models are the fastest, but I only recommend using them as a starting point. Try the fastest models first. They’re often the cheapest, and if they give you a great experience, bish bosh you’re done! If they don’t, try a slower model.
In the end, it’s all about creating a fantastic user experience, and that takes a level of taste only a human can apply.
