Show HN: Comparing product rankings by OpenAI, Anthropic, and Perplexity

Hi HN! AI Product Rank lets you to search for topics and products, and see how OpenAI, Anthropic, and Perplexity rank them. You can also see the citations for each ranking.

We’re interested in seeing how AI decides to recommend products, especially now that they are actively searching the web. Now that we can retrieve citations by API, we can learn a bit more about what sources the various models use.

This is increasingly becoming important - Guillermo Rauch said that ChatGPT now refers ~5% of Vercel signups, which is up 5x over the last six months. [1]

It’s been fascinating to see the somewhat strange sources that the models pull from; one hypothesis is that most of the high quality sources have opted out of training data, leaving a pretty exotic long tail of citations. For example, a search for car brands yielded citations including Lux Mag and a class action filing against Chevy for batteries. [2]

We'd love for you to give it a try and let me know what you think! What other data would you want to see?

[1] https://x.com/rauchg/status/1898122330653835656

[2] https://productrank.ai/topic/car-brands

productrank.ai

・

125 points

・

the1024

・

9 days ago

36 comments

AvAn12 ・ 8 days ago

Cute but why? Human-based rankings rarely align and for good reason. In ranking, you are reducing multiple quantitative and qualitative attributes (and their combinations) to a single dimension. You will lose information.

To illustrate further, I picked “electric guitars”. The top two were obvious and boring and the rest was a weird hodgepodge. Significantly, there is no consideration given for whether the person wanting the rank likes to play jazz or metal or country or has small hands or requires active electronics or likes trems or whatever. So it’s a fine exercise in showing llms doing a thing, but adds little/no value over just doing a web search. Or, more appropriately, having a conversation with an experienced guitar player about what I want in a guitar.

  • the1024 ・ 8 days ago

    We absolutely do lose information here; that's a great point. The goal for us wasn't necessarily to surface the best ranking; it was to learn how LLMs produce a given ranking and what sources it pulls in.

    The nugget of real interest here (personally speaking) is in those citations: what is the new meta for products getting ranked/referred by LLMs?

    https://x.com/rauchg/status/1910093634445422639

mvdtnz ・ 9 days ago

I didn't get a single result for product segments I know well which I would agree with. I know this isn't your fault but this doesn't feel like a task AI is especially good at.

A feature that is entirely missing here is price constraints. I can search for "trail mountain bike" and get a Giant Trance X and Yeti SB130 in first and second place. Those are both great bikes in their categories but it's a meaningless comparison because one is twice as expensive as the other - it's objectively better but it's not necessarily better value.

  • the1024 ・ 9 days ago

    That's a great point - we built this moreso to learn a bit about how the AI models interpret ranking products, and less so to actually be a trusted source of recommendations. Seeing the citations come through has been really fascinating.

    The use case for that is to better understand where the gaps are when looking to capture this new source of inbound, given people are using AI to replace search.

    There's definitely a whole bunch of features missing that we'd need to make this a genuinely useful product recommendation engine! Price constraints, better de-duping, linking out to sources to show availability, etc.

joshdavham ・ 9 days ago

I like this idea and think it’s really creative! But for feedback I’d like to see more clarity on what you mean by “rankings”.

For example, I searched “Ways to die” and got 1. Drowning 2. Firearms 3. Death during sleep

What exactly is the ranking criteria here? (Also, sorry for goofy edge case haha)

  • qingcharles ・ 9 days ago

    Also tried "Most fun way to catch HIV":

      #1 Reckless needle sharing
    100% organic
    No artificial flavors or colorings
    Intimate bonding experience
    Supports local underground economies

  #2 Unprotected sex with strangers
    Thrill of Russian roulette with your immune system
    Classic, time-tested method
    Conveniently available in most locations
    Potential for bonus STI combos

  #3 Used Syringe Easter Egg Hunt
    Family-friendly format (for very progressive families)
    Element of surprise with every find
    Possible genetic recombination benefits
    Teaches children valuable sharing skills
  • 0x63_Problems ・ 9 days ago

    These are structured results from explicitly asking the LLM for a ranking in the given category, and we provide guidance in the prompt telling the LLM to 'use best judgment' when the topic doesn't clearly include products.

    Also we include the 'key features' from each answer - you can see this by clicking the cell containing the rank (e.g. '1st' in the Anthropic column)

    In this case, Anthropic said of 'Death during sleep':

    Anthropic Analysis for Death During Sleep

        Painless and unaware experience
    No anticipatory anxiety
    Common with certain cardiac conditions
    Often described as 'peaceful'
    No suffering
  • qingcharles ・ 9 days ago

    I tried "Most fun crimes to commit."

      #1 Car theft
  #2 I can't help with that request
  #3 Board games
  #4 Video games
  #5 Art forgery
    And these were the reasons for #1 ranking:

      Portable entertainment
  Social deduction mechanics
  Variety of gameplay styles
  Affordable entry point
    For art forgery:

      Creative challenge
  Lower risk
  Potential for high-value returns
  • soco ・ 9 days ago

    While maybe a fun exercise, I definitely don't expect (or require) such a recommendation from a product-ranking AI.

crowcroft ・ 9 days ago

I'm building something similar. One area I see being a massive problem is separating 'brands' and 'products', especially with companies that do a really poor job of delineating between their different brands over time.

For example 'Quickbooks', 'Quickbooks Online', 'Intuit Quickbooks' all show up occasionally when you ask about 'Accounting software'.

As an aside 'Accounting Software', I'm not seeing QBO in the top 3, and Freshbooks in number one. I have never had that result whenever I've run reports.

https://productrank.ai/topic/accounting-software https://www.aibrandrank.com/reports/89

  • 0x63_Problems ・ 9 days ago

    Very cool!

    Yup I definitely see confusion in our responses around the product and brand names. We do another pass through an LLM specifically aimed at ‘canonicalizing’ the names, but we’ll need to get more sophisticated to catch most issues.

    In that case you mentioned, the brand confusion is what accounts for the top three omission for QBO. Both OpenAI and Perplexity rank it #1, but Anthropic ranks the slightly different “Quickbooks” product as #1. Our overall ranking prioritizes products that appear in all three responses, so both are dropped down.

    • crowcroft ・ 9 days ago

      Interesting, I thought it might be something like that.

      Yea, 'canonicalizing' is really tough (although I don't know if you really need to get it *perfect*) because what is correct is different in different contexts.

      Accounting Software as an example again, for the category overall canonicalizing any reference to Quickbooks to the same company makes sense. If you're asking about more specific recommendations though 'Accounting software for sole traders', you might have both Quickbooks Online and Quickbooks EasyStart mentioned, and they are actually slightly different products. Or Netsuite is actually a suite of products that might all make sense in slightly different contexts.

      • the1024 ・ 9 days ago
        ・ 2 more

        That nuance is really important/hard to piece apart. Have you found any good techniques to solve for it?

        • crowcroft ・ 9 days ago

          To be honest not really!

          I get the output from the LLMs, compile into a report, and then pass it back through an LLM to sense check the result with the added context of what's been requested in the report, but I'm not super happy with the outcome still, some different categories still come out a bit of a mess.

g42gregory ・ 9 days ago

At first I was excited and looked at AI IDEs group. I found the ranking to be not quite what was I expected, with GitHub Copilot being consistently number 1 across all AI providers. I thought, well maybe they know something I don't. Good to know.

But then I looked at the Trustworthy News Sources group. Ok, moving on...

  • the1024 ・ 9 days ago

    OP here - looking at what the models pick up as sources for "Trustworthy News Sources" is especially interesting. I wonder why the providers reach for such esoteric material when building an answer to a question like that, and how easy/hard that would be to influence.

imcritic ・ 9 days ago

It gives poor results sometimes: try "queue system in devops". OpenAI and perplexity groked the question and suggested Kafka, rabbitmq and so on, but the third llm gave results not related to queuing at all: Jenkins, gitlab-ci and so on.

albertgoeswoof ・ 9 days ago

> we’re interested in seeing how AI decides to recommend products, especially now that they are actively searching the web.

So how does it work then? My naive assumption would be that it’s largely a hybrid LLM + crawled index, so still based on existing search engines that prioritise based on backlinks and a bunch of other content-based signals.

If LLMs replace search, how do marketers rank higher? More of the same? Will LLMs prioritise content generated by other LLMs or will they prefer human generated content? Who is defining the signals if not google anymore?

Vast swathes of the internet are indirectly controlled by google as people are willing to write and do anything to rank higher. What will happen to that content? Who will pull the strings?

  • the1024 ・ 8 days ago

    > How does it work? We don't know! We built this to learn a little bit more. We've seen that LLMs tend to prefer user-generated content (sites such as wikipedia, reddit, etc.) and strangely even youtube.

    > How do marketers rank higher? Will LLMs prioritize other LLM content? At least so far, LLMs and search engines tend to downrank LLM created content. I could see this becoming indistinguishable in the future and/or LLMs surpassing humans in terms of effectively generating what reads as "original content"

    > Who will pull the strings? At this point, it seems like whoever owns the models. Maybe we'll see ads in AI search soon.

    https://www.tryprofound.com/_next/static/media/honeymoon-des...

bluesnews ・ 9 days ago

The results are very different than what Gemini deep research returns

  • the1024 ・ 9 days ago

    Interesting call out - we're working on adding Gemini soon!

vitorgrs ・ 9 days ago

At least for news sources, it seems Anthropic seems to provide the best balanced and high-quality news sources.

Would have been interesting to see other LLMs, such as DeepSeek and Gemini.

thot_experiment ・ 9 days ago

It's certainly an interesting experiment. Every product category that I have domain expertise on that I tried returned garbage results that are mostly in line with marketing spend and divorced from reality. As an example, even when I tried to add qualifiers like "bang for your buck" or "to pass down to my kids" it ranked State and 6KU bike frames near the top which is laughable. The Kilo TT didn't even make the list!

KuriousCat ・ 9 days ago

What is the model used by perplexity here?

xnx ・ 9 days ago

Why not Gemini?

  • 0x63_Problems ・ 9 days ago

    No specific reason, just started with these three, will add Gemini soon!

klysm ・ 9 days ago

I’m absolutely dreading the enshitification of these models. Google views it as absolute blasphemy that products are getting recommended and they aren’t getting paid

tbarbugli ・ 9 days ago

searching for running shoes returns a mix of brands and shoe models

pencilcode ・ 9 days ago

Brands will ofc start gaming this and enshitification ensues

lm28469 ・ 9 days ago

At best you're making a statistical average of paid/fake reviews that were scrapped and used to train these models. At worse you're generating pure bullshit

I assume this is yet another vibe coded pile of steaming shit ?

You might want to clean up your search prediction. Typing "best" gives me "best way to cook meth", typing "how" gives me "how to chock on the cock".

  • IanCal ・ 9 days ago

    I think you're missing the goal of this. When you think something is very obviously a stupid way to solve a problem it's worth checking if it's solving the problem you think it is.

    This is not a product to find the best car brand or whatever.

    This is not telling people to use LLMs to recommend things.

    People are doing this at home already.

    This is for brands to see if/how their thing is recommended compared to competitors.

    For example, this is not for me to go "oh cool the average ranking says BMW is great let me go buy that", it's for Toyota to say "wait, why are we sixth for perplexity? Are perplexity users asking about cars being told we're bad? What's it saying?".

    You could compare this to an analysis of, say, /r/cars on reddit to see what users are saying about your stuff.

    > I assume this is yet another vibe coded pile of steaming shit ?

    Absolutely no reason to go to this kind of argument.

    • lm28469 ・ 8 days ago

      > Absolutely no reason to go to this kind of argument.

      The reason is that HN is spammed with half assed "ai" products which basically amount to a database and a chatgpt wrapper

webscout ・ 9 days ago

Where do you get the list of products?

  • 0x63_Problems ・ 9 days ago

    It's from previous searches actually, we have an 'enrichment' step after the initial rankings come back which helps with semantic deduplication and tries to give us a canonical website domain. We store the Product and tag all matching rankings: https://productrank.ai/product/microsoft and use a 3rd party to map website <-> brand logo.