 All right, Wiz, today we go deep on DeepSeq. First question, is it really legit, this DeepSeq R1 model? I mean, in all senses of the word that I can think of, you could mean legit, yes. Yeah, it's kind of sent me personally down a rabbit hole trying to figure out what's really going on. and it's something like there's real rl happening you personally down a rabbit hole, trying to figure out what's really going on. And it's something like there's real RL happening here, instead of kind of some of the stuff we're used to, right? Let's say RL instead of SFT. It's not just supervised fine tuning sort of masquerading as RL. It's like legit RL that DeepSeq is using. Is that right? That's so right. Yes. It's really RL. It's not RL asterix. It's not RL being more generally forced into the SFT domain. It's really, it's doing the reinforcements and the learnings. Okay. Okay. And what we've seen so far in the field though, we're sort of alluding to this. It hasn't been like truly legit reinforcement learning. It's been this RLHF thing, right? it's been this reinforcement learning with human feedback and we've got to split this hair today don't we yeah it's it's really been fancy sft now that doesn't mean it's not valuable that doesn't mean strategies like dpo aren't useful for exactly what they are are said to be useful for right which is aligning the model to our preference but they're definitely still uh stapled to that sft domain in a way that prev just aligning the model to our preference. But they're definitely still stapled to that SFT domain in a way that prevents us from getting what we're gonna, we're gonna see as like the aha or Eureka moment during the training of these larger, you know, reinforcement learning approaches. Yeah, and when we talk about reasoning, we're talking about teaching the model basically to think, right? And so there's this real interesting idea at the bottom here where we can say, when we're not stapling outputs and demanding that they align with human feedback, the model can actually learn to, let's say, truly think in ways that are aligned truly with the universe, not just what humans think. Something like that is kind of where I get to. Is that a fair way to think about this as we head into today? Yeah, absolutely. The idea is exactly as you described, right? We're telling the model where the start and finish line are, right? But we're not telling the model how to get there. We're telling the model, figure out how to get there. Take whatever path you need, figure out a process by which you can get from start to finish reliably. And that's something that we don't do when we use things like RLHF and DPO. We instead say, hey, model, please do this. And stick as close to it as you can. And that's kind of like the magic of the approach that the folks at DeepSeek really popularized. And I guess it's not that surprising then that they're aiming for AGI, baby. They're aiming for AGI. Okay, Wiz, we're going to go ahead and have some discussions throughout, but it's time to kick off. Thanks for your insights. Welcome, everybody. We're going to talk DeepSeek R1 today, and we are going to try to go as deep as we can. There's actually just so much richness and depth at the bottom of all these different optimization algorithms. We don't have time to do all of them justice, but we look forward to having a rich discussion today and a little bit of a interesting story to tell along the way. If you want to jump in with your questions or in chat, please do. There's a link to the Slido. We will prioritize answering those questions at the end of today's event. But if we're having a discussion, feel free to jump in the live chat. All right, let's go ahead and kick off our second event in our large reasoning models series. This is our event on DeepSeek R1. So much goodness to come in reasoning in general. We're going to try to remind ourselves what we know today as we align our aim to the session and see if we can take it a step further. We really want to understand DeepSeq R1 today in context, in a proper technical context, building on what we've already learned about reasoning models. We want to try to take a wide angle view today of this reinforcement learning approach called group. what we've already learned about reasoning models we want to try to take a wide angle view today of this reinforcement learning approach called group relative policy optimization or grpo that was pioneered by the deep seek team and we want to talk and try to get some intuition about how it actually differs from something like proximal policy optimization and the optimization scheme used typically in RLHF implementations. So we want to take all of this and we also want to do some building shipping and sharing, of course. So shout out to the team at Unsloth. Those guys just continue to crush it, putting out amazing stuff. We're going to leverage their implementation of how to actually train our own version off the shelf of a Lama 3.1 model. So it can do reasoning. We can do this with a very, very small amount of GPU RAM today. It's amazing the technical advances we're seeing in LRMs. So what we want to do is we want to talk about DeepSeek as the company. Then we want to talk about DeepSeek R1 and kind of some of the way we got to where we are today before we do our own fine tuning. And we'll talk a little bit about this idea of distillation and how it differs from exactly what we're going to do today. All right, DeepSeek. The first thing we want to know about DeepSeek is that it's a competitor to OpenAI. Top line on DeepSeek, which I don't hate, is unravel the mystery of AGI with curiosity. Answer the essential question with long-termism. That's a new one for me, but I actually kind of love it. Let's learn a little bit more about the company. So, Leon Wenfang, he actually is a billionaire, the founder of DeepSeek, what is now known as DeepSeek. He founded a company in 2016, and he was doing stock trading with deep learning, with GPUs, and he was crushing it. you know, he's kind of ahead of his time. 2019, he founds High Flyer AI, a hedge fund, a hedge fund, really focused on AI trading algorithms. And that went really well, enough to make him a bunch of money. And in 2021, they really started to take to the next level with their Fire Flyer 2 lab that they were going to to stock with NVIDIA GPUs. They were trying to get like 10,000 A100s. And, you know, in this 2021 timeframe, this guy's already very well known in China. So he wrote the preface to sort of one of the biggest quant books from Jim Simons, uh or the man who solved the market is the name of the book. It's about Jim Simons and the quant revolution and sort of talking about this idea of quant trading. So very, very well known in the space, coming up in the right space at the right time. And, you know, so all of this kind of leads to 2023, there's like this bigger initiative to say, okay, let's go not just high, not just fire, we're going straight AGI. And that ultimately led to kind of incorporating DeepSeq. And since then, they've been shipping and sharing like a bunch of legends. We saw DeepSeq Coder, DeepSeq LLM, DeepSeq Mixture of Experts Model Series, DeepSeq Math from V1 to V2. And then in late 2024, we see DeepSeq R1 Lite Preview. Feeling quite familiar for those of us that have been following OpenAI, DeepSeq V3 and DeepSeq ultimately R1. So this has been a long time coming and it shouldn't be really surprising to folks in the industry to see that this is the kind of lab that came up with this. So, you know, putting all the headline stuff aside, this is really brass tacks, what was going on behind the scenes. This leads us into DeepSeek R1. And DeepSeek R1, I really like this quote from Jay Alomar, who put out the illustrated DeepSeek R1 recently, very, very worth your time to give that one a read. The same guy who put out the illustrated Transformer so long ago. It's the latest beat in the steady drum roll of AI progress. He goes on to say that, and we're going to kind of focus the session around this quote in some sense today. Like most existing LLMs, DeepSeq R1 generates one token at a time, except it excels at solving math and reasoning problems because it's able to spend more time processing through this idea of generating thinking tokens, which is very cool and Wiz is going to go into quite some depth on. And that helps us to deal with these chains of thought. So there's a number of pieces that we need to break down here. The first one is one we've seen before, this idea of chain of thought. This is the idea of thinking through something step-by-step. The original paper from the Google Brain Team in January, 2022, introduced this idea of chain of thought, a series of intermediate reasoning steps. And so chain of thought reasoning, you can see it's very good for things like simple math problems, or also it's very good turns out for coding problems. And there are many examples of the way this kind of went. It started off in 2022, looking at this sort of grade school math, Thank you. And there are many examples of the way this kind of went. It started off in 2022 looking at this sort of grade school math data set. Can we do well on it? It's like, yeah, we can do really well on it. And we're starting to reach the limits of just crushing it. And there's also this other sort of tangential idea that's related that's, I think, important to keep in mind as we use these systems. You know, there's also this other sort of tangential idea that's related that's I think important to keep in mind as we use these systems. You know, there's also this idea that we can think through something, we can also sort of refine our thinking about something. This idea of self-refinement came out in March 2023 in this paper, and it's just sort of the LLM is able to sort of be a judge to its own thinking. And this is, you know, quite llm as a judge as well and there's a lot of sort of core ideas here but you know you could imagine that you just are kind of having the llm think through and you're kind of having the llm judge some of the outputs and you're kind of having the llm do some self-refinement. And all of these kind of ideas are related to reasoning in some sense. They're related to these loops, or if you are familiar with us talking about agents quite a lot, this idea of looping through and thinking through things step by step. Those core foundational pieces have all led us to today. We have to fast forward quite a bit to sort of catch up. But this idea of being able to spend more time doing this is where we're kind of focused on as an industry today. In this paper from Google, again from deep DeepMind here in August last year, we can scale what we call test time compute now. And we can basically say if we think longer on difficult problems, the humans or the machines, we can probably do better. And this is the same idea we saw the following month come out from OpenAI when they released O1 in September. O1 thinks before it answers, these long internal chains of thought. This idea of test time compute is nothing but time spent thinking, right? So all of this is sort of core foundational stuff and what we what we see when we look at anybody's explanation is we see that they say things like through reinforcement learning, model A, B, C, D learns to hone its chains of thought and refine the strategies that it uses. In the spirit of trying to hone chains of thought and refine the strategies, we want to walk you through a set of demos. We're going to start with a simple off the shelf model we're going to see how it performs we're going to give it some chain of thought prompts and then we're going to see how it compares to something like deep seek r1 off the shelf it's time for some lm wizard in the house wiz passing it over to you oh yeah okay so, so this is the demo that we're going to do. The big thing that we want to focus on is the outputs that we're seeing, right? So when we talk about the ability to, you know, reason and everything like this this we kind of have to define you know what what we actually mean by reasoning etc uh what and what we mean in this case is the ability to work through problems in a way that uh you know emulates or or imitates uh the way that humans think so the idea is that we have this, you know, say, a classic fun toy question, right? How many R's in the word strawberry? When we ask the model, it doesn't really think about it, just says there's two. So, you know, there are two R's in the word strawberry, and also two other R's, making a total of two R's and two other R's for a total of two main R's. I don't even know what's happening, right? This is obviously a, you know, this is a great model. It's 3.370B, but it's just absolutely missing the point here. So what we do or have done to induce this kind of reasoning is something called chain of thought prompting, right? So chain of thought prompting is this think through your response step by step idea. Now, this is basically like a, like a crutch that we've used, right? It's a way to get the model to slow down a second, think through it, you know, don't just output whatever, whatever the first thought that comes to mind is right right? Just like you would hope to have normal people do, right? We're trying to induce this quote-unquote reasoning. And we get a different output, right? To find letters, number of R's in the word strawberry, let's break down the word into individual letters. We get strawberry, count the R's in the word. First R appears to position three, position eight, position nine, there are three R's in the word strawberry. Okay, so now, you know, it can do it, right, we have to do this like weird little prompting strategy, but it can get there. Okay, that's nice. What we want to do though, is we want to have a model that doesn't need that strategy, or more leverages that strategy to uh to expanded effect right and so that's where enters the scene uh deep seek r1 right so now you'll notice we don't have to provide this chain of thought style prompt we we certainly can and for math problems we we are told that we should but the idea is that you know we have how many R's in the word strawberry, and then we get thinking for 42 seconds. Now, that's a lot of thinking for this question, right? So there's no, make no mistake about it. That's, I would say, too much thinking for this particular question. But the idea is that we get this whole extremely interesting thought process, right? And some of this thought process helps us understand potentially, you know, why the model is making some of those off-the-cuff mistakes. So we have, I need to figure out how many times a little R appears. The word strawberry, let me start by blah, blah, blah, blah, blah, blah, blah. We get so much stuff. And then it goes, wait, that would be three R's, but I'm a bit confused because sometimes I might think of the spelling as strawberry, which has two R's at the end. Wait, no, let's check the correct spelling of strawberry. So we kind of can learn a little bit about the fact that, okay, well, it's just looking at berry, right? It's not yet looking at straw as part of strawberry, right? So, and you see it it kind of does this like weird doom spirally loop where it goes but maybe there's two r's and then it discovers again that there's actually three r's and then maybe there's two r's and then we go back finally to uh it's spelled like this there are three r's the answer is three uh the idea is there's and there's great comments about this in the chat right now, right? LLMs don't really think, chain of thought helps where we know the answer. This is largely true. The idea of chain of thought is essentially to fill up our context window with as much relevant context as we can so that we're able to pick out the correct answer from the pile, right? So it also helps us push a little bit deeper, a little bit further, and explore parts of a problem that, you know, that weren't explored at first blush. So again, you know, we can say all kinds of things. It's not really thinking. Okay, fine. Yeah. It's not really reasoning. That's probably, you know, somehow literally true in some senses, perhaps, you know, certain definitions, it is true. But the idea is that we're basically just blowing up our context window with things that we imagine are going to help us get to, or at least uncover a path to the correct answer. Now this is the most important piece of this whole puzzle, right? Is this idea that we're exploring the possible solution space and we're, we're therefore increasing the likelihood that we stumble upon the correct answer. So whether or not we want to call that reasoning, whether or not we want to nail that in as the model thinking is certainly up for debate. But I would say that this is, for the most part, how we're emulating that process. It is certainly how we're allowing the model to spend more compute at time of inference to increase the likelihood of getting the correct solution. Which is why this pattern fits in something called test time scaling or inference time scaling. Because it's allowing us to spend more resource outside of training and instead spend that resource at time of inference to stumble into the correct response. So with that out of the way, we'll go ahead and we will pass back to Greg who will explain us a little bit more of what's happening during the training process that lets this very fun, very useful effect emerge from our training. All right. Thanks, Wiz. Okay. So we've got this idea of chain of thought now and let's go a little bit deeper into the history of deep seek r1 and how it came to be i mean if we look back and we we sort of see okay the first primary thing that's worth paying attention to in the DeepSeq paper is this idea of GRPO. And we'll talk about that shortly, Wiz, we'll talk about that in the notebook. That policy optimization algorithm came from this DeepSeq math paper released in February last year. Now, this is interesting because DeepSeq math, titled Pushing the Limits of Math Reasoning in Open Language Models, is quite reminiscent of the path that we could follow that OpenAI took, right? So everybody's competing for the same thing. OpenAI October 2021, they were like, hey, we can solve math word problems. And it was like GSM 8K, got them. In February 2022, they were like, we could even solve harder math problems. And these are like more formal, like high school level math problems they explored and they found out, oh, okay, this method of expert iteration works pretty dope for this. We talked a little bit about this during the last session. Importantly, in May 2023, they released a blog and a paper titled Improving Math Reason with Math Reasoning with Process Supervision. And this gets to the point that we opened the event with today. So if you think about this process supervision idea, what we can do is we can look at each step of the reasoning and we can reward each step of the reasoning. Alternatively, we could look at just the outcome and we could reward just the outcome. Reading from the paper from OpenAI, in addition to boosting performance relative to outcome supervision, process supervision has an important alignment benefit. It directly trains the model to produce a chain of thought endorsed by humans. Okay, all right, that one's more HFE there, human feedback. We can train reward models to detect hallucinations using either outcome supervision or process supervision. Either way, we're looking at those COTs. And they said, yo, we crushed this math data set and we're releasing this other data. Enjoy. That was May 2023. DeepSeek comes in February 2024. Again, this is about open models. And they said, hey, our DeepSeq math, it's pretty small, but it's crushing it. And it's crushing exactly that same benchmark, the math benchmark. Let's take a look at what we can learn additionally from the paper as we dig into it. This is the place where they introduced group relative policy optimization, a variant of PPO, one that enhances math reasoning, while concurrently optimizing the memory usage of PPO. Now, this is really cool. And it's been speculated here on this last point, at least by the guys on the all in pod is like, they were resource constrained here. Fundamentally, I mean, they didn't have as much GPU compute to use as somebody like OpenAI. So, you know, it's a question of like, was the creativity and the innovation of this GRPO something that came from constraint or was it just a really great, cool breakthrough? It's not really clear one way or the other, but it does make a lot of intuitive sense to say, hey, we don't have enough memory to use. Let's figure out something to do that doesn't require as much memory to use. And, you know, this paper, I want to walk through the contributions of this paper, because they're kind of staggering, actually. And this first one, GRP, okay, dope. GRPO forgoes the critic model, instead estimating the baseline from group scores. There's a lot packed into that sentence, much more than we can get into today properly, but we're going to do our best to sort of cover it at a high level. Again, significantly reducing training resources. The idea here is connected in a lot of ways to a lot of different things. But essentially, it's the same idea we talked about with. We're not having the human feedback happening. Instead, we're using a true RL approach where we can look at a number of different kind of more objective views of analyzing how we think through problems like math problems, like code problems. Next, we demonstrate that GRPO significantly enhances the performance of our instruction tune model. Furthermore, we observe that enhancements in the out-of-domain performance are happening when we use this RL. Meaning, if we learn to think through stuff like math, stuff like coding, if we come up in the game as scientists, in other words, we can use our incredibly articulated thinking through ability in all sorts of other domains. And of course, we as engineers, we know this already. It's cool to see that AI can also do this. They go on to say we provide a unified paradigm. Okay. This is totally beyond the scope of today, but it's really cool. And I'm quite excited to dig in further to this, to understand the different methods, such as reinforced fine tuning, direct preference optimization, proximal policy optimization, and group relative policy optimization. Now you don't see RLHF in there, which is quite interesting. But finally, based on our unified paradigm, we explore the reasons behind the effectiveness of RL. So, you know, I would say these guys are pretty legit, the deep-sea guys. It certainly seems that way from reading some of their papers. And if we kind of zoom in on these ideas of using RL, enhancements in the out-of-domain performance during RL, unified paradigm to understand the different methods that use RL, the reasons behind the effectiveness of RL. Then we also have to deal with when we read something like the OpenAI blog through RL, O1 Learns to Hone Its Chain of Thought. Wiz will talk a little bit more about this in his notebook, but this is the image we get from the DeepSeek math paper. And there's a lot going on. The one thing I want to draw your attention to is that we get this sort of group of ways to reward our model in a GRPO. And this is something that we can sort of choose and we can design ourselves as we'll see when we do fine tuning with unsloth. And that's very cool. That is very cool. We still see KL divergence. We still see reference and reward model. There's a lot of similarities here. And, you know, if you guys want to see a follow up event to continue to go deeper into this, let us know. It definitely seems like something that further into the deep seek r1 paper that came out just this past month it takes this and it leverages this in a way that adds additional layers of complexity that we're going to try to highlight again as best we can here to go end to end today. We introduce our first generation reasoning models, DeepSeq R1, DeepSeq R10. DeepSeq R10 was a model trained via large scale RL without SFT as a preliminary step. And this is where we get into sort of how do you train models? And this is where there's just a lot that goes into models that get trained these days, whether you're talking about SLM type models that you can pull off the shelf, or whether you're talking about reasoning models, there's just a lot of steps that go into it. The easiest way to think about this is, here's the training pipeline. And this is from the illustrated DeepSeq R1, definitely encourage you guys to check it out. So we basically take DeepSeq V3, we do some supervised fine tuning, and then we do some fine tuning with RL. That's it, super easy to understand. When you start getting into the specific details is when it starts to get less easy to understand. For example, what are we using for supervised fine tuning datauning data exactly? Well, you probably won't be surprised to see that we're using reasoning data, COT-type examples like the ones that Wiz showed earlier. And then what we can do is we can sort of split up these two training pieces into this idea of like reasoning oriented RL and general RL. And all of a sudden we're in this space where now we have two different types of RL and neither one of them seem to be RLHF proper. And it's probably time to remind ourselves a little bit of how a typical model is trained today that we might be familiar with already. This is also from Jay Almar in their book on hands-on large language models. But I like this because basically you have to train the base model with some unsupervised pre-training. You have to do some instruction tuning to make it generally good at following instructions. And then you have to make it good at sort of understanding how humans want you to follow instructions. Be harmless, don't be toxic, etc. A way that we typically can think about this is by looking at our good friend the Shogoth where we train the big model. We then make it helpful and we can then make it harmless or we can dial in other aspect critiques now i want to go back again bring this back to what we were saying at the beginning if we think about instruction tuning in a classic sense this is fine tuning this is supervised fine tuning with instructions if we think about rlhf it really is also fine tuning with instructions. If we think about RLHF, it really is also fine tuning. It's just a finer fine tuning. It's really dialing this in. And we've covered PPO and the proximal policy optimization scheme for RLHF a number of times, we put in prompts that are maybe going to be a little bit challenging. And we want to make sure that we get a good response. It's not going to be harmful. It's not going to be toxic. So we might try to bait the LLM when we put in these prompts. And we're going to try to make sure that our policy model that we're training, that one is going to stay pretty close to the initial model through this check, this KL divergence check. The reward model is going to score the output. Again, it's going to score the output. It's not going to score each step of the chain of thought process, although we could do that as well. And then it's going to sort of decide on how it wants to change the value, change the state of how we're going to deal with a prompt like that in the future. We're going to go update the weights accordingly in our typically LoRa adapter. And then we're going to go through a number of iterations and we want to pump up these reward numbers. So we have RL, we have RLHF, we have rewarding the outcome, we have rewarding steps. Wiz, I want to bring you up to sort of have a little discussion here. I'm confusing myself again, right? It's like, are we using RL only in one of these? Why does this distinction really matter for us to understand right now? Yes, we are using RL in the case where we're talking about GRPO, the deep seek method. We are not using specifically RL for the RLHF pattern. This is largely due to the fact that what we're really doing at the end of the day is we're just doing SFT with extra signal, right? So now that good but there's no there's no real uh exploration that's occurring right we're just saying you know hey when the model produces a response we're gonna add some additional uh datums right to the to the final uh you know uh the the final weight update process, right? To try to be as simple as possible. The kind of magic of the... the final weight update process, right? To try to be as simple as possible. The kind of magic of the GRPO approach is that we don't care at all what the model outputs are as it relates to determining what the weight update should be, right? What we instead care about is we care about this kind of, you know, you know, as it relates to determining what the weight update should be, right? What we instead care about is we care about this kind of group, how well did it score on these metrics, right? And we're going to update based on that. And those metrics are not related to directly the, or necessarily directly the, the output itself. All that means is, you know, we care like that, that response is correct. We give score for that. We care that the response is potentially follows a specific format or prompt format. You know, we're going to, we're going to talk about that when we do the actual, when we do the actual build. But the idea is that we have this ability to say, hey, what we actually want to do is we want to explore this space unrestrictedly and not just say, hey, I noticed your generated sentence didn't score well, you know, do it again. It's more like saying, hey, you're scoring well on our metrics. I don't really care how you're getting there. You know, like you got there however you want. And why does it matter? Well, that exploration process is what allows us to uh have these kinds of moments high impact moments in training that uh that unlock quote unquote unlock uh abilities you know to to say the least so uh that's that's kind of where we're where we're at for that uh you know why it's important okay okay so yeah i like this idea of we're sort of allowing our model in a grpo scheme to sort of play right yeah you you know why it's important okay okay so yeah i like this idea of we're sort of allowing our model in a grpo scheme to sort of play right we're sort of allowing it to explore almost the environment if you will almost like the way you actually would think about an rl kind of system in its classic sense and instead of sort of imposing something that we say, this output is of the ultimate value, we're sort of saying, hey, figure out the answer. Whatever you do is dope, just get me the answer. And if the answer is right, good job. I like the sort of thinking about how you used to be graded in school and what is cheating and what is not cheating. I like the sort of idea, right? Where it's like, okay, if I give you just the right answer, is that enough? Well, GRPO kind of says yes, even if they got there a number of different ways, something like that. Correct. Whereas if you do explicit process supervision, it's kind of like when the teacher graded each step of your process. And even if it was right, you got the question wrong. Something like that, right? That's exactly it. Yes. Which is the worst, right? It is. The worst, yeah. We're going to go into this with an assumption, which is, which is we, we hold to be true, which is that if we get the right answer, most of the time, let's say that our process is working. You know, if we're able to arrive at the correct response more often than not, or in the case of math benchmarks, like more often than most other models can, then the process, something is inherently right with that process, right? We may not care to understand it or the way it looks might not be appetizing, et cetera, et cetera. But there's something that allows the model to get to the right answer about it. And that's the thing that we care about. Okay. And so is this Shoggoth? I mean, it feels like it's starting to break down for me a little bit here with a little smiley face on it. Like, is this still valid for you to think about this? I think the answer is that mask that we see, that smiley face that we see, is just a lot closer to the surface than I think I would have said pre this kind of like RL boom, where it takes so little effort to get to something nice, you know, from that. We just have to let the Shoggoth figure out how to interact. Yeah. We just throw at the food and the little bits. That's right. Just choose them up. That's right. Okay. Okay. And when it does that, the process of chewing those little bits up actually can be extrapolated to many other domains beyond math and code, which is cool. Right. Yes. Okay. All right. Excellent. Thank you, Wiz. And it's time to get into a little bit of fine-tuning, everybody. It's time to figure out how we're going to fine-tune our own reasoning model. We're going to use Unsloth today. We're going to fine-tune our own DeepSeq R1 style model. And shout out to the guys at Unsloth again. They're all about easy fine-tuning, training LMs faster. They are way down in the computation. We did an event covering more on Unsloth. Check it out to check out more about what they're all about. They joined us for that event. Why unsloth? Well, they're just really good at making stuff go burr with less. And we're going to see under the hood, still we have LoRa and Q LoRa. In fact, that was part of what they were able to do. They were able to take what other folks had done previously and make it work with QLaura and LoRa. And so we actually only need seven gigs of VRAM on our GPU to train our own reasoning model, which is big hype. Now note, and I really like this note from the blog, this isn't fine tuning DeepSeq's R1 distilled models, which they created a bunch of distilled models that they shot on the paper. And it's not using distilled data from R1 for tuning, although you can do all that stuff on Unsloth. This is literally taking any model off the shelf, like we're taking Lama off the shelf today and turning it into a fully fledged reasoning model. So with that, I want to give Wiz a little bit more time here. So maybe we'll save the discussion on distillation. If you're confused about what distillation is and what it means, a lot of times the way that we can easily think about it is we can sort of generate a lot of synthetic data from a big system, a teacher system, and then we can sort of use that synthetic data to train a smaller student system. And this is one way that we can think about distillation today with this synthetic data generation fine tuning. I want to again note that this is a little bit different than what we're going to see here. We're going to take our standard model, and then we're going to just do some straight up fine tuning with reasoning data. And that's the piece we're going to focus on using GRPO. We'll cover more on distillation and more on other ways to think about using and leveraging reasoning in our applications in future LRM events. So I want to bring your attention back as we head to Wiz to one of the key things you'll need to know to use these models today. And that is the idea of the thinking token. This is a game changer and something that's very worth paying attention to you, worth you paying attention to. Wiz, walk us through what we need to know to do SFT that actually induces reasoning on an off-the-shelf model today with these sweet thinking tokens. Over to you. Oh, yeah. Okay. So the did the the notebook so this notebook is uh is basically just extended from the unsloth notebook of the same uh of the same you know approach uh we're gonna check out some of the responses at the end from the unsloth notebook but the idea is is that uh yes uh we can do this in collab in a t4 instance uh as well as we can do it in a 100 instance as well as we can change hyper parameters uh basically this is all extraordinarily straightforward to do thanks to a number of innovations from people we love like unsloth like trl there you go okay so first things first we We have unsloth, like TRL. There you go. Okay, so first things first, we have Unsloth GRPO training. So this is not like exactly, exactly what happened in the paper, right? This is very close to what happened. The actual training volume is much different. The architecture is technically different, et cetera, et etc uh but this shows you how you can use this method right to train even like a llama you know in this case 3.18b model uh and in fact how we can do it with very little uh you know gpu power uh okay so What is the GPRO training process with RL? Basically, we have a few phases. Group sampling. Basically, what this says is that our policy is going to generate a batch of outputs. This is a method that we can kind of think about as doing some explore, right? So instead of just producing one output, we're going produce N outputs. And those N outputs are all gonna be very slightly different. And that very slight difference is kind of what allows us to explore the solution space as we move forward. Then we have, of course, our reward scoring. Our reward scoring is gonna be based on some reward functions that we have, which we'll talk about once we get to them in the notebook. And then we have our group-based advantage. Group-based advantage is, it's like thinking, how well does this group perform? And how well do the constituent members of the group perform relative to that right so the the idea is that since we're generating n examples let's just go with three right we're gonna have some average reward and then we're gonna assume that there's gonna be some number of responses that individual score will be above or below or or perhaps precisely, but more likely above or below that average. The ones that are above that average are considered advantaged and the ones that are below that average are considered disadvantaged. So the idea is that we're going to say, hey, these are the ones that were better than average, right? So we're going to build up a group of responses that are better than average. And then we're going to do some policy updates. Policy here is just a I don't know. right so we we're going to build up a group of responses that are better than average and then we're going to do some policy updates policy here is just a rl specific term but what we mean is the model the base model thing we're training the thing that we're trying to get to go uh burp right so the policy update is basically going to say hey these responses that were better than average let's be more like them and let's be less like these responses that are worse than average and then of course uh we do it again and then we do it again and then we do it again and then we do it again we do it many times in a row until we get some score that we are happy with right some some performance that we're we believe is desirable all right so we have this group-based approach uh it it avoids needing having that value head ppo situation right we're basically saying hey here are here are examples that we love uh we should we should be more like those and here are examples we don't love we should not be like those uh okay there you go so we have some imports we have a specific uh you know version of the TRL nightly this will be updated once they've locked in how they're doing their their sweet sweet GRPO unsloth has also provided a fast patch for the uh GRPO process once again I kind of spoke a little bit about it but we're generating like a bunch of responses right to the same thing because we're generating a bunch of responses to the same thing uh what we what we actually care about doing is we care about making sure that those bunches of responses right are generated quickly and so that's what this patch does um then we can load our model with regular old laura so again we're doing this with laura which is fun and exciting and interesting the idea with laura is pretty straightforward we have a uh we have a you know adapter that we're going to train the adapter is going to be what learns during this process this is part of what is necessary to make this you know possible to shove into a tiny notebook, of course. So this is how you can use the T4 if you would like to. But the idea is the same as every other LoRa. Fine-tune. Let's go. You love it. Okay. We're going to then use unslossed fast language model, which is basically just LoRa but fast. Okay, good job. Then we have to prepare our data. Now, this is the part that's a little bit spooky, right? Because we're still providing questions and answers, right? We still have a input and a target, you know? Like, that's what our data set looks like. Well, how is this not-tuning? We'll get to that. But the idea is that we still have, it's a very standard format dataset. We don't have some magical dataset. Now, if we're talking about supervised fine-tuning distillation methods, so we're talking about using SDG to generate reasoning traces and then using those to train another model sure that's fine uh but uh you know the idea here is that the the data set's pretty standard pretty stock uh you'll also notice that we have some xml uh chain of thought uh formatting as well as our system prompt which is going to contain a reasoning quote-unquote reasoning. And then we're going to have basically just an extraction process. Our data set just looks like this, right? We have some question and then we have some answer. We have some prompt that's been formatted. And then we have some content to format it with. That's basically it. The one thing that I want to make sure stands out to people is that there's absolutely no indication that the model should reason here, right? There's no, the prompt isn't saying reason about it. The, the, uh, the, uh, uh, you know, the specific answer doesn't contain like reasoning that we should emulate. Now, this is important to note because we have that process in the actual rl uh or the r1 model so not r10 we actually do this cold start that greg talked about right where we kind of we nudge the model a little bit at first to get like kind of a little bit good at this and then we let it play around right in this case we're just going straight to the playing around uh which is which is fun um and then we have bunches of reward models so there's this box right we're basically we're saying hey we need to we need to figure out like who's doing the best on what well this box is powered by these uh reward functions so these reward functions are basically just ways we can say if our if our process was doing well or not doing well if it was doing well we're going to give it cookies if it was not doing well we're not going to do it give it cookies and that's going to get us because we have a number of these reward functions that's going to be able to get us that average for each generated response and then a group average as well well. And that group average is what gets us the A, which is the advantage, right? Tells us, was it better than or worse than the average response in the pile? We have a number of these reward functions. There's the correctness reward function. And this is from work as done by a Twitter user, WillCCBD. you go. Great stuff. But the idea is we have correctness. Was it correct? We have some amount of like, was it an integer? Okay, that's good. Did the output fit the format we want? Exactly. And then did it fit it loosely? Do we have the correct number of XML tags? Right? And then finally, did we, this is just implementing the count XML reward pattern. But the idea is that we have like, in this case, we've got five different ways that we can record each or we can reward each generation based on what we are saying is loosely correct. Now that we have that, we are ready to start training. Our GRPO config looks like normal. You will notice there is not a lot of specific hyperparameters that we are caring caring about here this is all very stock uh hyper parameter instantiation you love it um there you go if you want to see the pretty graph you can change this to report to wand b that will produce that uh the the graph of where the line goes up and to the right as you train uh but otherwise we're we're not worried about it and then we can train and you'll notice in training that we get our reward kind of starts off excuse the massive scrolling starts off kind of whatever and then over time we slowly start to see higher and higher reward right now you would you would typically want to train this for uh you know 200 plus iterations which is going to take some number of hours in Colab. But the idea is that even with this, this, you know, limited subset of training, we can kind of see that this reward does slowly creep up a little bit as we start to learn, which makes sense. Now, you'll also see that we have these, you know know sample answers and these sample answers are very good because they help us understand how our model is uh how how the learning is going right so we have things like step you know steps uh in some of the responses we have things like you know these responses and our xml tags okay very cool you know we have these uh these attempts right and the idea is that as it trains it's going to get better and better at doing the things that we want because it's going to get scored higher and higher and higher and higher until eventually we get to the outputs we desire which are these kinds of long reasoning chains of answers i'm'm going to show you the outputs from the actual unsloth notebook, just because it's, they, they, they, the training went longer, so it looks a little bit better. We have this idea of the base model doesn't have a reasoning token, still very verbose, right? But it doesn't have a reasoning token. And then of course we have our, the, the fine-t, which has the reasoning tokens. And this is the, this is the thing that the model is learning to exploit these, learning to use these. The idea is that, you know, we, we can just hammer it into the model through this training process. Again, it doesn't take as long as like, you know, 20, 20 days, it still takes a little while. And the best part, the most fun part is that it does nothing for quite a long time. And then suddenly it starts getting better. And that is a special moment for anyone who likes training models. With that though, I'll pass you guys back to Greg, who can take us to some questions and close us up. Before I do, I got to remind you that we're here every Wednesday. Please don't forget to hit the little bell icon and subscribe, whatnot. We'll be doing more of these as the weeks go on every Wednesday. Catch us. Okay, there you go. All right, Wiz. Yeah, love to see that aha moment in training. Okay, let's wrap up everybody and head to some Q&A. What did we learn today? Well, we learned that DeepSeek has been in the game for a long time. DeepSeek R1 really came out of this tradition of high quality models coming out of an AI lab. Very similar to OpenAI. We see this sort of similar trajectory of the math and code and reasoning really come to fruition here in this R1 model. It uses RL, not RLHF, and uses less memory, perhaps because of the constraint of fewer GPUs. Unsloth, we saw, was a really great resource for us a shout out to those guys and then trailheads for you guys to explore more for me to explore more illustrated deep seek r1 shout out to jay alomar and the unified paradigm of all of these proximal policy optimization methods whether it's reinforcement, you know, learning, fine tuning or it's GRP, Oh, group relative policy optimization or classic ppo there really is something at the core of this that the deep sea guys are on to that i want to understand better okay so to wrap the magic is we already know chain of thought is good We already know that when we give the model steps, we can get better answers when reasoning is required. We know that DeepSeq R1 uses for real not just SFT or rather not just sft masquerading as rl like in rlhf it really do be rl in deep seek they taught it to play games with math and code and now it's pretty dope at lots of stuff it wasn't learning preferences okay it was learning like something true about the universe and it was learning specifically to do stuff in between the thinking tokens remember this ability is very close to the surface of llmsMs. It's pretty easy for us to bring out. Let's go ahead and wrap on that note and head to Q. We got some good ones, Wiz. We'll stay around for a couple extra minutes today. Jan Bors, is it true that because of the training process of DeepSeek, though through RL, no external data is needed, the model effectively trains by self-play. Biases are mitigated. Is it true that biases are mitigated by training through self-play? Probably not. But yes, this other stuff is true. You don't need, to be clear, you don't need like preference external data. You still need a target, right? You need like a question and an answer, say for instance, something that you can reward or determine the correctness of. So, you know, things like code, math are very easy to build verifiers for. Obviously you can get a little bit more complicated and build more complicated reward functions, but you don't need like external preference data or external chain of thought data, though the cold start implies that that is still useful, still helpful. That's right. Okay, so next question. What are the deep seek implications on startups and larger enterprises? I'm going to assume this means building, shipping and sharing production LLM applications, not the implications on, let's say, large GPU compute enterprises from the news. Like, how should we think about using these reasoning models like DeepSeq in our apps? We saw Strawberry, but where is it really useful? Yeah useful yeah i mean basically uh reasoning models are good at reasoning they're not good at other stuff though you know so it's like i would say right now they fit in places where you want that enhanced reasoning so for planning task decomposition stuff like this but for like the actual just work a day you know function calling like function colleagues not even supported in in the deep seek model yet uh you know you're you're probably there there's some frameworks that get around that et cetera et cetera but the yeah yeah it's good at reasoning yeah it's like it's like a phd right when do you hire a phd right it's like you don't hire them for everything in fact they're they're out of the running of many jobs because they're too educated, right? It's like, you don't hire them for everything. In fact, they're out of the running of many jobs because they're too educated, right? You know, it's like, so, you know, do you really need something that needs all this deep thought or just do you just need something that's going to just do the job, boom, you know? And I think we really have to ask this question more and more, which is super interesting, especially as we get into sort of multi-agent systems. And then we have the PhD guy on the team and we have the lower level individual contributor and the manager, and we've got all these different roles and maybe we choose different models for them, is sort of my intuition here. So, okay. A B asks, what's the most effective way to detect and monitor LLM hallucinations in reasoning models? They do be hallucinating. Basically, the same stuff that we used before just applied to the final result. Inside the reasoning chain, they hallucinate like mad the whole time. That's almost part of their charm, right? That's almost part of why they work. so you know detecting the hallucinations inside the reasoning chain probably is a fruitless uh endeavor uh but the final result yes we still want to you know whatever faithfulness style metric you you most prefer uh we want to make sure that final result is is uh important yeah yeah yeah like do you want you want your team member to come up with crazy ideas and you throw most of them away? You know, it's like, we're in a lot of questions about hallucinations. I think this is going to be a very interesting thing that enterprise has to deal with is when to use these and how to constrain them well. What are potential use cases for deep seek 1.5 b uh okay so this is just a small deep seek model this isn't even a reasoning model um i don't know do you have any opinion on deep seek models in general beyond the r1 model uh the distilled ones are like much worse at reasoning than the then the big boy right like for sure it's not it's not even close uh but uh there's still they're still better than uh than models that don't reason that reasoning you know it's like i i hate to be being a broken job but like if you were using a a 1.5b model to do some kind of reasoning workload first of all that that's brave all the power to you but second of all like deep seek 1.5b will be better however i i would still say like the the place the unlock here exists in those higher uh parameter count models that's where we get kind of like the real the juice is worth the squeeze right uh i i would say i i would be hesitant to recommend to to to use deep seek 1.5b for like heavy reasoning tests for things like specialized uh tasks so a a a by now famous example uh of the uh of this is like the countdown game right where you have you know five numbers and you have to build an equation that gets you the sixth target number right this very specific task can be handled very well by a very small model so if you have instances like that which are very focused hyper specific reasoning tasks I would say that's where we're looking to see things like deep seek 1.5 B style models really shine yeah and just to just to clarify for the audience here That's what we're looking to see things like DeepSeq 1.5b style models really shine. Yeah. And just to clarify for the audience here, that is an R1 model. That's a distilled version of QN of 1.5b that we're talking about, DeepSeq 1.5b. Okay. A couple of quick rapid fire, and then we'll wrap here, Wiz. One, is it feasible to use these in rag chains uh yes but why is sort of my intuition is that right uh yes you can use it in rag i just wouldn't it's not very good at following instructions it's not really you know you but you can uh there are a number of workflows where it does make sense to use this agentic workflows uh you know all kinds of fun you know bits and bobs you but you can uh there are a number of workflows where it does make sense to use this agentic workflows uh you know all kinds of fun you know bits and bobs you can do and and create and leverage that reasoning style but for things that are very direct where like rag is a knowledge task right it's not like a reasoning task really and truly uh for most cases rag is just like get the right context in the context window, model's going to crush it, right? If you have a RAG system that really relies on reasoning for whatever reason, then perhaps you'll see slight improvements, especially as we get better instruction following variants coming out the pipeline. But for now, I would say it is of course it's feasible okay alright alright final technical question how is prompting reasoning models different from prompting chat models yes so this is this is great uh no clue we're still learning uh this is a this is in flux uh we are we are not yet there uh you know deepek provides some guidance on how to prompt their model. Other people are learning strategies for open AI as closed-source reasoning models. Again, we're kind of having to rethink how we approach prompting these models. We certainly don't always need to think step-by-step,'s still useful, it turns out, right? We don't always need to be focused on the same prompt styles that have worked before. We got to do some of our own exploration in order to figure that out. But yeah. All right. All right. Yeah, we are. We are. We're all trying to learn. And I mean, I don't know the answer to that question either. Go play with it. Play with it all the time. You know, and this gets to the last question here is the excitement around DeepSeq warranted and, you know, do its advancements justify the massive sell off in the market is another sort of aspect of this. But I mean, I think, yes, and I think we talked about this before, where sort of reasoning is kind of having its alpaca moment here. We're sort of seeing that we're getting this distribution of reasoning capabilities throughout the entire community. Now with Unsloth, that's equally as exciting to me as the DeepSeek thing. Although I don't have any skin in the game in terms of specific stocks. So, you know, I mean, is it warranted for you all of the excitement around Deep seek maybe not all of it from my opinion but but much of it uh it seems like maybe the technical community you know sort of is has aligned more with the public a little bit more on this than than i would have expected but i think it's warranted for us as builders what do you think i have i you know i have all kinds of thoughts about this but i'm not a market expert i'm not a financial expert i my my job isn't markets and stuff uh lots of great readings you can do on on the the impact that deep seek did or didn't have on the on the market that uh you're you're welcome to read and synthesize reasoning is dope though and more coming LL is coming up soon, right? Okay. The model technically is something to be excited about and the paradigms of making RL easier and more aligned with what we're trying to do with models is hype, no matter what happens with all the other stuff. Yeah. That's right. That's right. Awesome, Wiz. Well, thank you so much for your expertise today and teaching us how to do that fine tuning on reasoning. We'll see you next week for Coconut. Make sure that you guys join us next week as we continue our Reasoning Model series. We're going to be talking continuous chains of thought with Coconut. And we're We're also going to be within the next six weeks, launching the next cohort of the AI engineering bootcamp. And, you know, it's rapidly evolving space, even in our current one, we've got to add vision language models. We've got to add some reasoning. So it's very, very exciting time to get in on the game. There's never been a better time. If you're interested in accelerating your career, check it out. Otherwise, stay tuned because following next week, we also have a very special event on cursor that will be exactly what any of you guys that are just starting to get into the game need to know to get up and running to build ship and share your very first ai engineering application so that's going to be a real fun one join us for the next couple of weeks on youtube live join us in the discord community to build ship and share with all of the legends there we're meeting weekly on monday mornings you can meet directly with me and many others in the Discord to talk about what you'll be building, shipping, and sharing this week. Thanks to everybody that joined us on YouTube. Thanks to everybody that joined us on LinkedIn. Until next time, keep building, shipping, and sharing, everybody, and we'll certainly do the same. Have a great week, and we'll see you soon. Bye, guys.
Deepseek-R1 & Training Your Own Reasoning Model
DeepSeek is dominating global app stores, but what’s behind its latest breakthrough? Join us as we dive into DeepSeek-R1, the first Large Reasoning Model (LRM)—how it was trained, how it compares to OpenAI’s o1/o3 and Gemini Flash Thinking, and what it means for the future of AI reasoning.

We’ll break down the multi-stage RL training, distillation process, and key takeaways from the DeepSeek-R1 paper. Don’t miss this deep dive into the next wave of reasoning!

Join us every Wednesday at 1pm EST for our live events. SUBSCRIBE NOW to get notified!

Speakers: 
​Dr. Greg, Co-Founder & CEO AI Makerspace
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO AI Makerspace
https://www.linkedin.com/in/csalexiuk/

Apply for The AI Engineering Bootcamp on Maven today! 
https://bit.ly/AIEbootcamp

LLM Foundations - Email-based course
https://aimakerspace.io/llm-foundations/

For team leaders, check out!
https://aimakerspace.io/gen-ai-upskilling-for-teams/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/z96cKbg3epXXqwtG6

#deepseek 

00:00:00 Exploring the Legitimacy of Deep Seek R1 Model
00:04:30 Understanding Deep Seek R1 in Technical Context  
00:09:09 Innovations in AI: Understanding Deep Seek R1  
00:13:39 Evaluating Off-the-Shelf Models with Chain of Thought Prompts
00:17:45 Exploring Chain of Thought in AI Models  
00:22:10 Process Supervision vs. Outcome Supervision in AI Training
00:27:00 Understanding Reinforcement Learning Methods
00:31:22 Training AI to Follow Human Instructions  
00:35:41 Unrestricted Model Exploration and High-Impact Training
00:40:18 Exploring DeepSeek R1 and UNS Sloth Innovations
00:44:59 Understanding GPU Power in RL Training Process
00:48:59 Preparing Data for Fast Language Models
00:53:03 Understanding Stock Hyperparameter Instantiation  
00:57:18 Understanding Deep Seek's Role in Reasoning Models
01:01:20 Detecting and Monitoring Language Model Hallucinations in Reasoning Models
01:05:31 Differences Between Prompting Reasoning Models and Chat Models