The issue I have had with llms since the start has been the slop. Contrary to popular belief, the open web was mostly slop before ChatGPT. We just used to call it SEO blogspam. And all llms are trained on them.
So when I tried google's Deep Research in Gemini, I ran into the same problem. It was basically a typical llm chat response, but a lot more verbose and citations to the same blogspam listicles that makes regular human research difficult.
How do I avoid this pitfall? So what I am asking is, how do I use deep research?
20 comments
Use it for what? If your goal is understanding… probably don’t. If you’re trying to produce a document for an audience that doesn’t really care and will just use an LLM to summarize it without any fact checking, then give the minimal prompting to get the token count you need. This is where LLMs excel —- sequences of tokens that are plausible use of human languages where the cost for verifying if they have semantic value exceeds five or ten minutes.
My naive, yet practical, view is that all LLMs produce output that reflects what was most commonly found on the internet. So if your area of interest has a lot of slop, then that is exactly what Deep Research will return.
So if you want to do "real deep research" into some topic, then you start with the various summaries and drill down. I use Google Scholar to locate papers, then I search for the cited papers. It is tedious, but the only way that I know which avoids the pitfalls of random regurgitation of easily found materials.
Currently AGI is marketing hype. Critical thinking with real understanding, i.e. knowledge building hasn't yet been automated. You need to put in the intellectual effort.
That sounds almost exactly what ChatGPT's Deep Research does. (Not to be confused with Grok's which seems to be doing mostly busywork)
It's a bit like a crawler, goes through papers, then follows down to other relevant papers. One advantage I see it seems to be able to search through movies as well.
But in the end you get a stack of reading materials and still have to do the work of reading them. I did try to get it to summarize them into instructions but it drops the main points.
What topic or field are you researching? Is it something academic, programming related, or...?
It generally helps if you ask it to focus on specific kinds sources (gov sites, academic, etc.) but it really depends on the topic. Easy for some topics (the general sciences) but not so easy for other ones (popular culture stuff or current events).
Market research.
I guess I should have included the purpose of my research.
I think it's going to be terrible at that :( Separating SEO spam from useful products isn't a strength of LLMs.
Probably a normal prompt without deep research would be better for that (but less recent), especially for topics that humans previously discussed a lot in the past (like on pre-AI reddit). Signal to noise ratio is only going to go down, down, down from here... :(
I don't think you'll get useful results from deep research in a case like this, trying to distinguish between all the contemporary advertisers and cut through the spam. Deep research is good at summarizing sources, and it works well if you can tell it what kind of trustworthy sources to focus on. It's not so good at judging the relative merit of each source. And in this case, there are no trustworthy sources... it's all just spam. If you manage to get good results out of it for market research, you should probably just start a business and become a billionaire selling that functionality instead of whatever you were researching :)
For market research, Exa.ai may be a better option. It's a search engine that crawls the web and index pages in LLM embeddings. For my searches, it delivers good results that are not found in Google's first page because the SEO spam is burying it. You should give it a try.
Hey, co-founder of linkup.so here. Did you try our API? I'd love to understand why you chose one option over another
Hire an expert in the field who can create 100 sentences of "dont do this... dont do that..." then create your own business as a middleman who keeps this list a top secret and supplies the service to others.
Then in a couple months go out of business when someone tricks the AI to spill its prompt
I am actually trying to use it for the purpose it is built for; Commercial research.
The one thing I had to do is ask for a References section at the end of the research. You can then take that and go through it to check things.
Integrate our web search agent (linkup.so) which has been trained to cut through the noise :)
Searching for free coupons :)
[dead]
Do actual research like a researcher instead of being a lazy bum with LLMs?
That seems to be only real course of action, but I need to be sure if I am doing anything wrong.