Supervised fine tuning on curated data is reinforcement learning
arxiv.org

・

71 points

・

GabrielBianconi

・

3 days ago

19 comments

anndvision ・ 3 days ago

We recently ran similar experiments and saw that fine-tuning small models on automatically curated high-quality outputs from a large model can beat large-model performance while reducing inference costs by up to 30x and inference time by up to 4x.

We benchmarked closed-source (OpenAI, Google) and open-source (Qwen) models on multi-turn maze navigation (BabyAI), agentic RAG (Multi-Hop), and agentic tool use (τ-bench).

We're still running a few experiments and plan to update the post with additional results in a few days.

Looking forward to trying out importance weighting soon!

Curated Behavior Cloning: Small LLMs Can Beat Large Ones at 5-30x Lower Cost: https://www.tensorzero.com/blog/curated-behavior-cloning-sma...

  • TheTaytay ・ 3 days ago

    Thanks for this - I’ve spent the last hour reading your docs and blog. I like the primitives you’ve exposed in your APO, and particularly like the decision to separate out the structured inputs from the prompt when you record an LLM call, so I can finally perform optimizations and evals on past calls.

    Quick question : you mentioned unsloth in the blog post. Which of the fine tuning providers mentioned is using unsloth under the hood?

iandanforth ・ 3 days ago

How is this kind of analogy helpful? You can frame any optimization problem as RL if you try hard enough. RL is a method of optimization which calls the optimum "reward maximization". You can craft the reward function any which way you want.

The key point about RL is that it is a sequential decision making process. If you don't have something (an agent) making multiple decisions over time while interacting with an environment, then why bother calling it RL?

  • imtringued ・ 3 days ago

    I personally am quite disappointed by the abstract:

    "Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting."

    uh no? SFT is maximizing the RL objective in a dense reward setting. The entire point of RL, specifically actor-critic and Q-Learning, is that the RL method turns the sparse reward into a continuous dense reward against which a model can be trained on with classic gradient descent.

    I mean look at the definition of Q-Learning and the bellman equation it uses. It maximizes the current reward by choosing the current action based on whether it maximizes the predicted reward, not the actual reward, which doesn't have to be continuous or produce a gradient. You can build an RL based maze solver where only the goal gives a reward to the model and it would work, albeit it would train extremely slowly.

    Meanwhile supervised fine tuning always produces a continuous gradient on every single token.

    • chongliqin ・ 2 days ago

      TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training.

      We here make a connection to REINFORCE style policy gradients which would not show any of the behavior you mentioned above.

henriquegodoy ・ 3 days ago

It's cool to see the perspective that many problems (somekinda communication problems, look at lawyers, compliance and etc...) can be solved by treating AI less as agents and more as modular components within a larger system. Once we build a working process—monitored through evals—we can then reduce costs by distilling these modules. That means starting with superintelligent models and later distilling them down to just a few billion parameters, instead of needing hundreds of billions.

mandevil ・ 3 days ago

Interesting to see two independent researchers on this. Makes me curious as to what the back-story is? Side project?

  • jtspringenberg ・ 3 days ago

    Author here, just to clarify: we are both no longer working for DeepMind. This was purely an independent effort for the sake of research and understanding! Happy to answer any questions.

  • babelfish ・ 3 days ago

    Especially interesting given they both work for Google DeepMind.

markisus ・ 3 days ago

Something seems off with equation (5).

Just imagining Monte Carlo sampling it, the middle expectation will have a bunch of zeros due to the indicator function and the right expectation won’t.

I can make the middle expectation be as close to zero as I like by making the success threshold sufficiently high.

  • chongliqin ・ 2 days ago

    Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this doesn't affect the bounds as the constant is the same across equations. Will update the paper to incorporate.

stolencode ・ 3 days ago

> For example achieving 66.7% on the AIME 2024 dataset.

We worked _really_ hard, burned _tons_ of cash, and we're proud of our D- output. No wonder there are more papers published than actual work being done.

  • supermdguy ・ 3 days ago

    That corresponds to a 10/15, which is actually really good (median is around 6)

    https://artofproblemsolving.com/wiki/index.php/AMC_historica...

    • stolencode ・ 3 days ago

      Isn't the test taken only by students under the age of 12?

      Meanwhile the model is trained on these specific types of problems, does not have an apparent time or resource limit, and does not have to take the test in a proctored environment.

      It's D- work. Compared to a 12 year old, okay, maybe it's B+. Is this really the point you wanted to make?

  • jpcompartir ・ 3 days ago

    This is a nonsense critique.

    Modest results are worth publishing, as are bad results.