Gemini 3 Deep Think drew me a good SVG of a pelican riding a bicycle

> What if the models have just gotten really good?

Kagi Assistant remains my main way of interacting with AI. One of its benefits is you're encouraged to try different models.

The heterogeneity in competence, particular per unit in time, is growing rapidly. If I'm extrapolating image-creation capabilities from Claude, I'm going to underestimate what Gemini can do without fuckery. Likewise, if I'm using Grok all day, Gemini and Claude will seem unbelievably competent when it comes to deep research.

Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.

Why frame it as rigging? I assume they would teach the models to improve on tasks the public find interesting. Then we just have to come up with more challenges for it.

Simon has a private set of SVG tests he uses as well. He said that the private ones were just as impressive.

> For those claiming they rigged it.

I don't think they "rigged" it, but might be given a bit more push on that part since it's going for a very long time now.

Another benchmark is going on at [0]. It's pretty interesting. A perfect scoring model "borks" in the next iteration, for example.

> Rant: This is why AI is going to take over, folks are not even trying the least.

It might be drawing things alright, at least some cases. I seldom use it when my hours long researches doesn't take me to the place I want, and guess what? AI can't go there, either. It hallucinates things, makes up stuff, etc. For a couple of things I asked, it managed to find a single reference, and it was the thing I was looking for, so it works rarely in my cases.

Rant: This is why people are delusional. They test the happy path and claims it knows all the paths, and then some.

[0]: https://clocks.brianmoore.com/

and it will be folks using AI taking over for at least a while...

Some people try, most people don't.

AI makes doing almost anything easier for the people that do..

Despite the prophesied near-term obliteration of white collar work, I've never felt luckier to work in software.

Everyone should have their own private evals for models. If I ask a question and a model flat out gets it wrong sometimes I will put it in my test questions bank.

It is visually outstanding. The only thing that sticks out to me is that the steering column bends out forwards towards the ground (negative trail), which would make it oversteer rather than self-stabilize. Interestingly there's a slight positive trail bend in the second one, though.

Agreed, good is quite an understatement. Every item is drawn superbly, and the basket with the fish is just great. Feels like a big jump over the other models (though granted, this is such a known "benchmark" by now, it's likely gamed to some extent).

> a great bicycle

It's not. Sorry.

Go look at some real bicycles for reference.

Parallel hypothesis: the intensity of competition between models is so intense that any high-engagement high-relevance web discussion about any LLM/AI generation is gonna hit the self-guided self-reinforced model training and result in de facto benchmaxxing.

Which is only to say: if we HN-front-page it, they will come (generate).

There was Lenna for digital image compression (https://en.wikipedia.org/wiki/Lenna).

A pelican on a bike is SFW, inclusive, yet cool.

It is not a full benchmark - rather a litmus test.

You can just try other svgs, I got some pretty good ones.

(*Disclaimer: I work for Google, but also I have zero idea about what they trained deepthink on)

So, again, when the indicator becomes a target, it stops being a good indicator.

note that this benchmark aside, they've gotten really good at SVGs, I used to rely on the nounproject for icons, and sometimes various libraries, but now coding agents just synthesize an SVG tag in the code and draw all icons.

I think it could still be an interesting benchmark. Like, assuming AI companies are genuinely trying to solve this pelican problem, how well do they solve it? That seems valid, and the assumption here is that the approach they take could generalize, which seems plausible.

Is that cheating, or is that just working smarter not harder?

read it aloud. “since it’s an FAQ”, where FAQ is pronounced “eff-ay-queue”

Source: