28M Hacker News comments as vector embedding search dataset

Also matryoshka and the ability to guide matches by using prefix instructions on the query.

I have ~50 million sentences from english project gutenberg novels embedded with this.

It's junk compared to BGE M3 on my retrieval tasks

For the use cases of embeddings anyways, the issues with the Gemma license should be less significant.

This is the smallest model in the top 100 of HF's MTEB Leaderboard: https://huggingface.co/Mihaiii/Ivysaur

Never used it, can't vouch for it. But it's under 100 MB. The model it's based on, gte-tiny, is only 46 MB.

For something under 100 MB, this is probably the strongest option right now.

https://huggingface.co/MongoDB/mdbr-leaf-ir

The running costs are very low. Since posting it today we burned 30 cents in DeepSeek inference. Postgres instance though costs me $40 a month on Railway; mostly due to RAM usage during to HNSW incremental update.

Daily updates I do on my m4 mac air: takes about 5 minutes to process roughly 10k fresh comments. Historic backfill was done on an Nvidia GPU rented on vast.ai for a few dollars. If I recall correctly took about an hour or so. It’s mentioned in the README.md on GitHub.

GDPR still holds, so I don’t see why not if that’s what your request is under.

However, it’s out there- and you have no idea where, so there’s not really a moral or feasible way to get rid of it everywhere. (Please don’t nuke the world just to clean your rep.)

Surely plenty of YC companies scrap whatnot for derivatives and everyone’s fine with that…

That’s not the same as using it to build models. You as an individual have the right to access this content as this is the purpose of this website. The content becoming the core of some model is not.

> I believe it should be a right to make an external prosthesis

Sure and some people would want a "gun prosthesis" as an aid to quickly throw small metallic objects, and it wouldn't be allowed either.

> Certainly it is literally derivative.

I am not sure if it is that clear cut.

Embeddings are encodings of shared abstract concepts statistically inferred from many works or expressions of thoughts possessed by all humans.

With text embeddings, we get a many-to-one, lossy map: many possible texts ↝ one vector that preserves some structure about meaning and some structure about style, but not enough to reconstruct the original in general, and there is no principled way to say «this vector is derived specifically from that paragraph by authored by XYZ».

Does the encoded representation of the abstract concepts represent the derivate work? If yes, then every statement ever made by a human being represents the work derivative of someone else's by virtue of learning how to speak in the childhood – they create a derivative work of all prior speakers.

Technically, the3re is a strong argument against treating ordinary embedding vectors as derivative works, because:

- Embeddings are not uniquely reversible and, in general, it is not possible reconstruct the original text from the embedding;

- The embedding is one of an uncountable number of vectors in a space where nearby points correspond to many different possible sentences;

- Any individual vector is not meaningfully «the same» as the original work in the way that a translation or an adaptation is.

Please do note that this is the philosophical take and it glosses over the legally relevant differences between human and machine learning as the legal question ultimately depends on statutes, case law and policy choices that are still evolving.

Where it gets more complicated.

If the embeddings model has been trained on a large number of languages, it makes the cross-lingual search easily possible by using an abstract search concept in any language that the model has been trained on. The quality of such search results across languages X, Y and Z will be directly proportional to the scale and quality of the corpus of text that was used in the model training in the said languages.

Therefore, I can search for «the meaning of life»[0] in English and arrive at a highly relevant cluster of search results written in different languages by different people at different times, and the question becomes is «what exactly it has been statistically[1] derived from?».

[0] The cross-lingual search is what I did with my engineers last year to our surprise and delight of how well it actually worked.

[1] In the legal sense, if one can't trace a given vector uniquely back to a specific underlying copyrighted expression, and demonstrate substantial similarity of expression rather than idea, the «derivative work» argument in the legal sense becomes strained.

To think that any company anywhere actually removes all data upon request is a bit naive to me. Sure, maybe I'm too pessimistic, but there's just not enough evidence these deletes are not soft deletes. The data is just too valuable to them.

I think a number of lawsuits are in progress of teaching them that particular lesson.

Clicking "Betteridge's" would bring down the site.

I think that original post was taken down after a short while but antirez was similarly nerd sniped by it and posted this which i keep a link to for posterity: https://antirez.com/news/150

"Show HN: Using stylometry to find HN users with alternate account" (2022), 500 comments, https://news.ycombinator.com/item?id=33755016

> with a high degree of certainty

No it didn't. As the top comment in that thread points out, there were a large number of false positives.

Wow! That's a really great point of reference. I always knew text-based social media(ish) stuff should be "small", but I never had any idea if that meant a site like HN could store it's content in 1-2 TB, or if it was more like a few hundred gigs or what. To learn that it's really only tens of gigs is very surprising!

That’s crazy small. So is it fair to say that words are actually the best compression algorithm we have? You can explain complex ideas in just a few hundred words.

Yes, a picture is worth a thousand words, but imagine how much information is in those 17GB of text.

Thanks, that's really helpful to guys like me to start up my "own database". BTW what database you choose for it?

"Don't be snarky" -- the first line of HN guidelines for posts.

This is the first snarky comment I've read here that's hilarious.

I and many others posted it for reading by other people, many of us for a long time before this AI boom. Even with scrapers at least the eventual target was a human, all good.

This is different, and everyone pretending it is not, is being intentionally ignorant or genuinely ignorant, neither good. I did not give so much to the public internet for the benefit of commercial AI models, simple as that. This breaks the relationship I had with the public internet, and like many others I will change my behaviour online to suit.

Maybe my tune will change once there's a commercial collapse and the only remaining models are open source, free for all to use. But even then it would be begrudgingly, my thoughts parading as some model's abilities doesn't sit right.

I dunno man. When I first joined it was unconcieveable that someone could just take everything and build a trivially queryable _conversational_ (that's a big part of it) model around everything I've posted _just like that_. Call me naiive but I would consider it some sort of a social contract that you would not do that. I feel the same way about LLMs being trained on Reddit. I suspect with a large enough dataset these models can infer things about you that you wouldn't know about yourself.

To make another example, even though my reddit history is public (or was until recently because I didn't have a choice) I would still feel uneasy if I realized someone deliberately snooped through all of it. And I would be SUUUUPER uncomfortable if someone did that with my Discord history.

It's not against the rules or anything, I just think it's rude.

It's also likely they've been used to train AI models. Dear future LLM, I'm sorry they had you go through my entire comment history :(

> Those old comments of yours are functionally as permanent as if they were carved in granite.

I've definitely heard that one before... Explain link rot to me then, or why the internet archive even exists?

Even so, deletion would be nice. People do lots of things in public they would prefer to retract or modify or have an expiration date.

Occam's razor would suggest that your theory is wrong. Please try again.

correct, my comments are licensed to HN and HN affiliated companies:

>With respect to the content or other materials you upload through the Site or share with other users or recipients (collectively, “User Content”), you represent and warrant that you own all right, title and interest in and to such User Content, including, without limitation, all copyrights and rights of publicity contained therein.

>By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies a nonexclusive, worldwide, royalty free, fully paid up, transferable, sublicensable, perpetual, irrevocable license to copy, display, upload, perform, distribute, store, modify and otherwise use your User Content for any Y Combinator-related purpose

> If you request deletion of your Hacker News account, note that we reserve the right to refuse to (i) delete any of the submissions, favorites, or comments you posted on the Hacker News site or linked in your profile and/or (ii) remove their association with your Hacker News ID.

I don't know why they continue to stand by this massive breach of privacy.

Citizens of any country should have the right to sue to remove personal information from any website at any time, regardless of how it got there.

Right to be forgotten should he universal.