Extracting AI models from mobile apps

Boils down to "use Frida to find the arguments to the TensorFlow call beyond the model file"

Key here is, a binary model is just a bag-of-floats with primitively typed inputs and outputs.

It's ~impossible to write up more than what's here because either:

A) you understand reverse engineering and model basics, and thus the current content is clear you'd use Frida to figure out how the arguments are passed to TensorFlow

or

B) you don't understand this is a binary reverse engineering problem, even when shown Frida. If more content was provided, you'd see it as specific to a particular problem. Which it has to be. You'd also need a walkthrough by hand about batching, tokenization, so on and so forth, too much for a write up, and it'd be too confusing to follow for another model.

TL;Dr a request for more content is asking for a reverse engineering article to give you a full education on modal inference

Just having the shape of the input and output are not sufficient, the image (in this example) needs to be normalized. It's presumably not difficult to find the exact numbers, but it is a source of errors when reverse engineering a ML model.

they have a very "interesting" definition of private data on the paper. it's so outlandish that if you buy their definition, there's zero value on the trained data. heh.

they also claim unsuppervisioned users typing away is better than tagged training data, which explain the wild grammar suggestions on the top comment. guess the age of quantity over quality is finally peaking.

in the end it's the same as grammarly but without any verification of the interested data, and calling the collection of user data "federation"

In principle, device manufacturers could make hardware DRM work for ML models.

You usually inference those on GPUs anyway, and they usually have some kind of hardware DRM support for video already.

The way hardware DRM works is that you pass some encrypted content to the GPU and get a blob containing the content key from somewhere, encrypted in a way that only this GPU can decrypt. This way, even if the OS is fully compromised, it never sees the decrypted content.

`enc(coffee cup) == enc(donut)` would be an interesting guarantee.

I'm a mobile developer and I'm new to using Frida and other such tools. Do you have any tips or reading material on how to use things like Frida?

this would be yellow in https://en.wikipedia.org/wiki/Spiral_Dynamics but we are still a mix of orange and green.

Standard corporate hypocrisy. "Rules for thee, not for me."

If you actually expected anything to be open about OpenAI's products, please get in touch, I have an incredible business opportunity for you in the form of a bridge in New York.

They got backlash, but (if I'm not mistaken) it was ruled that it's okay to use copyrighted works in your model.

So if a model is copyrighted, you should still be able to use it if you generate a different one based on it. I.e. copyright laundry. I assume this would be similar to how fonts work. You can copyright a font file, but not the actual shapes. So if you re-encode the shapes with different points, that's legal.

But, I don't think a model can be copyrighted. Isn't it the case that something created mechanically can't be copyrighted? It has to be authored by a person.

I find it weird that so many hackers go out of their way to approve of the legal claims of Big AI before it's even settled, instead of undermining Big AI. Isn't the hacker ethos all about decentralization?

> reference to the prevalent argument that training AI on data, and then offering it via AI is being a parasite on that original data

Prevalent or not, phrased this way it's clear how nonsense it is. The data isn't hurt or destroyed in the process of being trained on, nor does the process deprive the data owners from their data or opportunity to monetize it the way they ordinarily would.

The right terms here are "learning from", "taking inspiration from", not "being a parasite".

(Now, feeling entitled to rent because someone invented something useful and your work accidentally turned out to be useful, infinitesimally, in making it happen - now that is wanting to be a parasite on society.)

The fact that models creators assert that they are protectrd by copyright and offer licenses does not mean:

(1) That they are actually protected by copyright in the first place, or

(2) That the particular act described does not fall into an exception to copyright like fair use, exactly as many model creators assert that the exact same act done with the materials models are trained on does, rendering the restrictions of the license offered moot for that purpose.

LLMs are trained on works -- software, graphics and text -- covered by my copyright. What's the difference?

If I understand the position of major players in this field, copyright itself is optional (for them at least).

Is there a material difference between the copyright laws for software and the copyright laws for images and text?

Yeah no.

An example for legal reference might be convolution reverb. Basically it's a way to record what a fancy reverb machines does (using copyrighted complex math algorithms) and cheaply recreate the reverb on my computer. It seems like companies can do this as long as they distribute protected reverbs separately from the commercial application. So Liquidsonics (https://www.liquidsonics.com/software/) sells reverb software but puts for free download the 'protected' convolution reverbs specifically the Bricasti ones in dispute (https://www.liquidsonics.com/fusion-ir/reverberate-3/)

Also, while a SQL server can be copyright protected, a SQL database is not given copyright protection/ownership to the SQL server software creators by extension of that.

> My next article will be about CoreML on iOS, doing the same exact thing :)

Can't wait - thanks for writing it up!

The weights are my training data. I scraped them from the internet

I think the comment author means offering inference via Firebase, with the model never leaving the backend.

This works, just like ChatGPT works, but has the downside of 1. You have to pay the computing for every inference 2. Your users can't access it offline 3. Your users will have to use a lot of data from their mobile network operator. 4. Your inference will be slower

And since SeeingAI infers the model every second, your and your customers bill will be huge.

Exactly. Put another way, tensorflow is not an AI. You can build an AI in tensorflow. You can also resize images in tensorflow (using the traditional algorithms, not AI). I am not an expert, but as I understand, it is common for vision models to require a fixed resolution input, and it is common for that resolution to be quite low due to resource constraints.

> LLMs are not massive archives of data.

Neither am I, yet, I am still capable of reproducing copyrighted works to a level that most would describe as illegal.

> And before you knee-jerk "it's a compression algo!"

It's literally a fundamental part of the technology so I can't see how you call it a "knee jerk." It's lossy compression, the same way a JPEG might be, and simply recompressing your picture to a lower resolution does not at all obviate your copyright.

> I invite you to archive all your data with an LLMs "compression algo".

As long as we agree it is _my data_ and not yours.

Copying a single sentence verbatim from a 1000 page book is still plagiarism.

And is technically copyright infringement outside fair use exceptions.

It doesn't matter. It's still a derived work.

When I was at school, we were sometimes all sat down in front of a TV to watch some movie on VHS tape (it was the 90s).

At the start of the tape, there was a copyright notice forbidding the VHS tape from being played at, amongst other places, schools.

Copyright rules are a strange thing.

copyright refers to the act of copying the material at hand (including distribution, reproduction, performance) etc.

as an example: saying “i really like james holden’s inheritors album for the rough and dissonant sounds” isn’t covered by copyright.

if i reproduced it verbatim using my mouth, or created a derived work which is noticeably similar to the original, that’s a different question though.

in your example, a derivative work example could be akin to only quoting from the book for the audience and modifying a word of each quote.

“derived” works are always a grey area, especially around generative machine learning right now.

[dead]

*only available in the USA, terms and conditions apply.

most other places use fair dealing which is more restrictive https://en.m.wikipedia.org/wiki/Fair_dealing

Easy to claim, harder to justify once you start charging money for your subsequent creation.

Unless all LLM are a ruthless parody of human intelligence, which they may be, the legal issues will continue.

The moment you earn money from it, that's not fair use anymore. When I last checked, unlimited access to said models were not free, plus it's not "research" anymore.

- Addenda -

For the interested parties, the law states the following [0].

Notwithstanding the provisions of sections 17 U.S.C. § 106 and 17 U.S.C. § 106A, the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include:

    1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
    2. the nature of the copyrighted work;
    3. the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and
    4. the effect of the use upon the potential market for or value of the copyrighted work.

So, if you say that these factors can be flexed depending on the defendant, and can be just waved away to protect the wealthy, then it becomes something else, but given these factors, and how damaging this "fair use" is, I can certainly say that training AI models with copyrighted corpus is not fair use in any way.

Of course at the end of the day, IANAL & IANAJ. However, my moral compass directly bars use of copyrighted corpus in publicly accessible, for profit models which undermine many people of their livelihoods.

From my perspective, people can whitewash AI training as they see fit to sleep sound at night, but this doesn't change anything from my PoV.

[0]: https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors

as a human being, and one that does music stuff, i don’t download terabytes of other peoples works from the internet directly into my brain. i don’t have verbatim reproductions of people’s work sitting around on a hard disk in my stomach/lungs/head/feet.

LLMs are not humans. They’re essentially a probabilistic compression algorithm (encode data into model weights/decode with prompt to retrieve data).

I don't see how this is a double standard. Comparing a person interacting with their culture is not comparable in any way. IMHO, it's kind of a wacky argument to make.

Human creators don't store that 'influence' in a digital machine accessible format generated directly from the copyrighted content though.

Although with the 'good new everyone, we built the torment nexus' trajectory of AI my guess is at this point AI companies would just incorporate actual human brains instead of digital storage if that was the requirement.

Can you clarify this a bit. I presume you are talking about the tone more than the implied statement.

If the last sentence were explicit rather than implied, for instance

This article seems to be serving the growing prejudice against AI

Is that better? It is still likely to be controversial and the accuracy debatable, but it is at least sincere and could be the start of a reasonable conversation, provided the responders behave accordingly.

I would like people to talk about controversial things here if they do so in a considerate manner.

I'd also like to personally acknowledge how much work you do to defuse situations on HN. You represent an excellent example of how to behave. Even when the people you are talking to assume bad faith you hold your composure.

I don't seem to be able to edit it, apologies I will try not to let this type of thing get to me in future.

I would also like to point out that this is a fine tuned classifier vision model based on mobilenetv2 and not an LLM.

Please don't cross into personal attack or otherwise break the site guidelines when posting here. Your post would be fine with just the first sentence.

https://news.ycombinator.com/newsguidelines.html

[flagged]

Can you please edit swipes out of your HN comments? Your post would be fine with just the first sentence.

This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

Yes nothing wrong with cool software or showing people how to use it for useful things.

Sorry I'm just kind of sick of the whole 'kool aid', 'rage against AI' thing a lot of people seem to have going on and the way is presented in the post. I have family members with vision impairment helped by this particular app so its a bit personal.

Nothing against opening stuff up and understanding how it works etc. I'd just rather see people build/train useful new models and stuff with the open datasets / models already available.

I guess AI kind of does pay my bills in a round about way.

> AI models are intellectual property

If companies train on data they don't own and expect to own their model weights, that's hypocritical.

Model weights shouldn't be copyrightable if the training data was pilfered.

But this hasn't been tested because models are locked away in data centers as trade secrets. There's no opportunity to observe or copy them outside of using their outputs as synthetic data.

On that subject, training on model outputs should be fair use, and an area we should use legislation to defend access to (similar to web scraping provisions).

Going a step further, weights, i.e. coefficients, aren't produced by a person at all – they're produced by machine algorithms. Because a human did not create the weights, the weights have no author. Thus they are ineligible for copyright in the first place and are in the public domain. Whether the model architecture is copyrightable is more of an open question, but I think a solid argument could be that the model architecture is simply a mathematical expression – albeit a complex one –, though Python or other source code is almost certainly copyrighted. But I imagine clean-room methods could avoid problems there, and with much less effort than most software.

IANAL, but I have serious doubts about the applicability of current copyright law to existing AI models. I imagine the courts will decide the same.

I think you have to distinguish between a model, its implementation, and its weights/parameters. AFAIU:

- Models are processes/concepts, thus not copyrightable, but are subject to trade secret law, contract and license restrictions, patents, etc.

- Concrete implementations may be copyrighted like any code.

- Parameters are "facts", thus not copyrightable, but are similarly subject to trade secret and contract law.

IANAL, not legal advice, yadda yadda yadda.

The weights are a product of a mechanical process, 5 years ago it would be generally uncontroversial that they would be not subject to copyright in the US... but 'industry' has done a tremendous job of spreading confusion.

Datasets want to be free.

Actually, in terms of copyright control "The Federal Circuit went on to clarify the nature of the DMCA's anti-circumvention provisions. The DMCA established causes of action for liability and did not establish a property right. Therefore, circumvention is not infringement in itself."[1]

https://en.m.wikipedia.org/wiki/Chamberlain_Group,_Inc._v._S...

>Circumventing a copy-prevention system without a valid exemption is a crime, even if you don't make unlawful copies. Copyright covers the right to make copies, not the right to distribute; "doing what you want with it yourself" may or may not be covered by fair use. Whether or not model weights are copyrightable remains an open question.

If that is the law, it is a defect that we need to fix. Laws do not come down from heaven in the form of commandments. We, humans, write laws. If there is a defect in the laws, we should fix it.

If this is the law, time shifting and format shifting is unlawful as well which to me is unacceptable.

Disclaimer: As usual, I anal.

Your comment confused me, but I'm very interested in what you're getting at.

> Circumventing a copy-prevention system without a valid exemption is a crime, even if you don't make unlawful copies.

Yep, this is the DMCA section 1201. Late '90s law in the US.

> Copyright covers the right to make copies, not the right to distribute

This is where I got confused. Copyright covers four rights: copying, distribution, creation of derivative works, and public performance. So I'm not sure what you were getting at with the copy/distribute dichotomy.

But here's a question I'm curious about: Can DMCA apply to a copy-protection mechanism that's being applied to non-copyrightable work? Based on my reading of https://www.copyright.gov/dmca/:

> First, it prohibits circumventing technological protection measures (or TPMs) used by copyright owners to control access to their works.

That's not the letter of the law, but an overview, but it does seem to suggest you can't bring a DMCA 1201 claim against someone circumventing copy-protection for uncopyrightable works.

> Whether or not model weights are copyrightable remains an open question.

And this is where the interaction with the wording of 1201 gets interesting, in my (non-professional) opinion!

just imagine, like just for a second how it becomes illegal to train anything that does not then afterwards produce, if publicly used or distributed, a copyright token which is both in the training set - to mark it - and in the produce - to recognize it.

so this is where it all goes in several years, if i were the gov.

Is using millions of copyrighted works to train your AI a valid exemption? Asking for a few billionaire friends.