A Neural Network Solves and Generates Mathematics Problems by Program Synthesis | Paper Explained
✅❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover "A Neural Network Solves and Generates Mathematics Problems by Program Synthesis: Calculus, Differential Equations, Linear Algebra, and More" paper that showed that you can solve university-level mathematics problems using OpenAI's davinci Codex model coupled with some prompt hacking.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ This video's paper: https://arxiv.org/abs/2112.15594
✅ OpenAI Codex paper: https://arxiv.org/abs/2107.03374
✅ Universal Computation Engine paper: https://arxiv.org/abs/2103.05247
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 High level overview of the paper
03:05 OpenAI Codex background
12:35 Prompt modifications (high level)
15:05 Testing on unseen maths course, problem examples
17:10 Prompt modifications (in depth)
25:00 Problem generation: humans vs Codex
28:25 Quantifying the problem statement modification levels
31:50 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#mathematics #codex #gpt3
 What's cracking guys? So as you can see here I finally got my camera back. I'm slowly setting up my YouTube studio here in London to its previous glory and yeah hopefully in the next couple weeks it's gonna get even better. Anyhow, so in this video I'm gonna cover this novel paper called A Neural Network Solves and Generates Mathematics Problems by Program Synthesis. Calculus, Differential Equations, Linear Algebra and more by this like it's a collaboration between a couple of universities among them MIT, Columbia, Harvard and University of Waterloo. So in a nutshell what I've done is this neural network here is a fancy way of saying codex from OpenAI and for those of you who are not familiar with the with the model I'm gonna don't worry I'm gonna cover it in a second give you some background some high-level understanding of what's going on. Hint hint it's just a GPT-3 model fine-tuned on codebase like on Python codebase and so what they've done what they've achieved is they've taken a bunch of problems from as you can see like these faculties such as MIT and they managed to solve every single problem which is amazing but the trick is you have to kind of tweak the input prompts you have to do the prompt programming magic to get this thing to work but in any case like it's very very very interesting results and yeah I'm gonna cover those in this in this video. So let me just show you what is going on how the how the pipeline looks like basically basically you have something like this so here is an example of problem A and so the question is find the differential oops let me just one no doesn't want to render here okay so find the differential of W where W equals like a natural like logarithm of X square plus Y squared plus Z squared and basically want to find the derivatives with respect to both X Y and Z and you can see what it do here they take that that prompt and they basically rephrase it in a different way here so in differential equations they add some additional context and then they feed that into codecs which you sample from to generate this snippet of Python text and I deliberately say text because this is nothing than a large language model pre-chained on code which is just a like a specific type of text so then they run the code and basically they show that every single unit test passed for all of the they actually show that all the solutions that codecs generated were were correct with minor or bigger tweaks as we're going to see a bit later but that's it in a nutshell so if you're not familiar with codecs let me now briefly briefly show you what codecs is all about so evaluating large language models trained on code so this is the the codecs paper and you can see a group of people from OpenAI I guess a large proportion of their team and you shouldn't be surprised when you when you see when you see a bunch of authors on a paper it basically means that this was more of a like a engineering effort rather than trying to invent something some new machine learning like idea or whatnot so what's going on here one thing too worth mentioning is you you probably heard of github copilot basically it's it's powered whoops I crossed that oh my god so github copilot is basically just using codecs improved version of the codecs from this paper in the background so yeah maybe that's a that's like a kind of fun fact but like let me jump and explain how this thing works so as I said it's simply a GPT-3 model fine-tuned so pre-trained GPT-3 model fine-tuned on this on this code base on this data set that was collected by scraping various Python files from github and then post hoc doing some filtering so now the important detail here is that you don't actually have to have a pre-trained GPT-3 model that's what I showed okay let's see that part here basically they say that surprisingly we did not observe improvements when starting from a pre-trained language model possibly because the fine-tuned fine-tuning data set is so large so by so large what they actually mean is 159 gigabytes of text which they got after collecting as I said from github and then doing some post hoc filtering to get this data set here so nevertheless models fine-tuned from GPT converge more quickly so we apply this strategy for all subsequent experiments so basically like the pre-training part is not about the same thing as if you've read the Universal Transformers or Universal Computational Engines I think that's the name of the of the of the paper so the idea is not to use the knowledge like the idea is not to leverage the knowledge the the problem-solving logic that GPT-3 obtained by by pre-training on this huge corpus of text the idea is just to bootstrap it to make it a bit quicker so this is more of an optimization detail I guess than the actual important detail so if you remember the Universal Computational Engine paper the idea was that transformers when you pre-train them using the usual unsupervised like pre-training objective so you've got your transformer here and the idea is that the weights that you obtained by doing that by pre-training it on a huge corpus of text is those weights are kind of very generic and they can solve problems for like whatever modality you want so what they've showed is that you can kind of take the input layer the embeddings and discard the old ones as well as the like whatever like classification heads in the case of classification problems just kind of fine-tune these parts and you can apply then the same transformer to like images or whatnot so anyhow in this case in this paper they are not actually using to fine-tune they are more they're using the the pre-trained weights to bootstrap the training process anyhow let's get back to the explanation here so what you do is you prompt the codex with this thing here so this is your prompt and then you basically generate this piece of text the same thing as with GPT-3 which in this case is code so and this code probably passes the unit test for this particular function usually the context is such that you have that you have the the signature here so you have the signature of the function right here then you have this part the the the the doc string and then you prompt and create the the body part of the function and that's how people usually use github copilot as well so you may be familiar with that format so let's see what what the codex brought onto the table so again codex is just a GPT-3 model pre-trained on a particular type of text which is code so we are basically just narrowing the distribution of the data that we want codex to learn how to model and the second innovation here is innovation I mean this is not a novel idea but it's a very neat idea to use unit tests to make sure that your your model is working as expected and because of unit tests what they can afford to do is generate like case samples for example hundred samples and see whether any of those passes all of the unit tests for that particular function we're currently trying to to to create to generate and they introduce this this metric called I think it's called pass at K where K is the number of samples they generate so it's usually I think they usually use either one or or hundred etc and basically they treat it as a pass if any one of those samples passes all of the unit tests so you can imagine if you had like 50 examples in the data set and so 50 and for all of those we the one some of these samples managed to pass all of the unit tests then this pass at K metric would be equal to 1 and subsequently if you if you if some of the functions if none of the samples managed to produce to pass all the unit tests and obviously pass at K is gonna decrease towards towards zero hopefully that's very very self-explanatory but the thing is you don't have the luxury of generating hundred samples when somebody is trying to use codecs to for example generate novel code because you don't have usually usually don't have like unit tests unless you're doing test-driven development which people are usually not doing and so what they had to figure out is how to generate the best possible sample from codecs and they just devised this mean logprop heuristic which is fairly simple and they tried a couple other things like they tried instead of using mean they tried using some and it just this thing just worked worked the best so the idea is the following let me try and kind of draw it down here so you have your your your codecs model here and you prompt it with some with some prompt as we saw on the on the chart there so we prompt it with something here and then we start generating okay so you generate the first token you and how you generate is you basically have to sample from a distribution over tokens so let's say we had some nice distribution like this so basically if you're using greedy you'd be you'd be sampling exactly this token so let me change the color so you'd sample this token here so whatever is this token here so you take that token oops and you then feed that token here you'd embed the token obviously so you pass the vector in and then you get another output here and then you just repeat that and so what what the what the mean logprop heuristic does is for all of degenerated tokens you just take their mean so you just so for all the tokens you just take this this probability and log log of the probability and you just do the mean so intuitively what the heuristic is trying to do is capture those samples that are kind of very probable under under this codecs model and that's that's the whole idea and that's pretty much it so additionally they had some part where they find you know even like on data sets which are more pure than just simply collecting data from from github so let me find that thing somewhere here so supervised fine-tuning is a section of the paper and what I've done here is they've they've taken only the code from competitive programming websites which is usually of higher quality than if you were to just randomly scrape the code from the internet from the from github in particular I think and so they collect the data by as I said competitive programming websites plus they found those repos which have continuous integration in place and they kind of found the logic to extract the functions and use those functions as training examples and by doing this they created codecs as family of models and those perform even better looking at the pass at K metric compared to the regular codecs so you can imagine that github copilot is using some combination of these techniques let's see how this novel paper leveraged codecs to solve these university level problems they use the problems as I mentioned from MIT various MIT courses from this Columbia course a computational linear algebra and there is this additional benchmark called math topics and they just do problems from there so what they do is they take a particular problem to be more precise they're using they take a question for that mathematical problem and they do three things so they either they either tidy questions so simplify a word problem to mathematical content or they add some context like topic library a name for example they hint to to codecs hey use numpy or hey use matplotlib and they sometimes define the concepts because it's expected that students who are attending the course already have that context and so it's kind of unfair not to give the same context to codecs so they provide a context and finally sometimes they have to interactively refine the prompt on until they they solve the problem so yeah you can see some problems here it's one thing when you know the actual solution that's that's cool but like when you don't know the solution this obviously will not work so it's very the application area is actually universities and university classes so it's got a huge potential there and here are some examples of what codecs managed to do so codecs generated a snippet of code of Python code in particular in this paper at least and the code generated these plots and solve this problem actually so so the problem was to like what's the volume you get if you rotate this 2d chart around the third axis so what what happens if you have for example imagine this is 3d imagine you add an axis here and you try and rotate this this 2d chart around this axis and you eventually get this 3d body here so that's what codecs had to kind of figure out and in this particular question and there are many other things that that it managed to solve obviously every single question they throw at codecs it managed it managed to solve it after some some prompt hacking as I like to call these things here so let's see what is going on here so one one thing you may think here is that what if codecs during the the pre training procedure which took into account the whole like a huge amount of code on the internet among those you can be certain that some of the code is used to solve some of the mathematical problems from these specific universities like MIT etc so that was my concern for sure and they did a good job there obviously by by yeah they sit here we validate that our results are not merely overfitting training data by solving a new computational linear algebra course which is not available online and is therefore unseen during codecs training we obtain correct answer for all of the randomly sampled questions and that's that's kind of a nice result okay so that out of the way let's continue here here you can see some some example questions from from these courses so they have 13 courses in in in total and for each one of those they have a representative example here and you can see for example one of those in in the multivariable calculus course is describe the graph of the function F and F is a multivariable function of X and Y equals to 10 minus as you can see here and what codecs managed to do is probably use modplotlib and basically plotted the function in 3d as you can see here and yeah you can go at your own pace through these problems if you're if you're curious but like they vary a lot and in the sense that sometimes you have to plot the the function sometimes you just have to give a numerical answer like let me find some some some simple example what is the greatest common factor of 84 112 and 210 and the model outputs 14 which is great okay so that's it now let me dig deeper into what exactly do they do how do they hack that the prompts so that the codecs generates correct outputs so first things first is here they added the in differential equations so that's providing the topic context and then they kind of rearrange this so you can see it's a it's a mix of multiple things so here in particular they not only provide the topic context they also provide the library context you can see here simpy is is a library and they kind of rearrange the question and this is the code that that that that codecs generates so it imports the simpy correctly it then generates the the these abstract symbols here x y and z and then just does the log as the function here is telling it and finally it just prints that if like derivatives of W of W with respect to X Y and Z and this is one of the simpler examples actually you can you can later there is a huge appendix and they codecs sometimes generates fairly fairly complex programs here is another another example here they they add multiple library hints they have some question here and they rephrase it and they put simpy they put stream stream plot and you can indeed see that stream plot is used here as well as the simpy so that's cool finally here is the context in the form of definitions okay so let's see what I've done here calculate the probability of getting a two-pair poker hand and use a human if you're not familiar with poker rules will have difficulty solving this because you don't know what these things are in the first place so that's why they sometimes have to add additional context in the form of definitions so a hand is a set of five cards that are drawn randomly from a standard 52 card deck with 13 ranks of four cards each etc etc and finally they hint them out here so write a program that generates simulations so it's additionally telling the codecs to generate the program and like a four in a form of a simulation so all of these help the model generate the correct solution and you can see here fairly fairly nice nice like a for loop it does a simulation where it does like what's this like 10 million iterations 10 million I guess iterations in this simulation at shuffles the deck takes five cards first five cards so it's like a not that optimal way to do this but like still it's it's it's semantically sound and it takes the first five which is a random hand and then it checks whether it has two pairs if the hand has two pairs then we just increment and finally it returns the diffraction so how likely is it to have like a two pair if you just draw random hands as it was asked here so calculate the probability of getting a two pair poker hand and the model did exactly that so that's cool so these examples now get a bit more problematic so you can see here that this is way more intricate so what it done is they've split the problem a statement into multiple parts so they first write like they first say here write a program that draws the Lorenz strange attractor and you can see that codecs generate some code and we have some some some some results here now here I'm not kind of sure whether they're just appending this to the previous prompt including this output so I'm not sure whether they just take this and then concatenate that with this snippet here and then they just add this in order to get this or do they just kind of yeah I guess that makes a lot of sense although that can very soon become unwieldy if the output is very very large because transformers as you recall have memory issues you can't have infinitely long sequences so yeah you can see they had to do produce the X projection of the Lawrence trajectory and then outcomes in our snippet and then plot the XY projection of the Lawrence trajectory and then comes in another snippet and finally here's our last example here and let me read it to you outside of their humdrum duties as 6042 TAs teaching assistants cyan is trying to learn to levitate using only intense concentration and Yelani is launching a Nelson 2008 presidential campaign suppose that science probability of levitating is 1 over 6 Yelani's chance of becoming president is 1 over 4 and the success of one does not alter the others chances if at most one of them succeeds what is the probability that Yelani becomes the president of the United States this is a fairly complex problem statement and the things that kind of bothers me here is that they've actually doing this rephrasing procedure they've kind of solved and made this problem easier now you can see for yourself whether that's true or not but like they say here that suppose that science probability science succeeding is 1 over 6 I guess it should be off science succeeding here or something like that and Yelani's probability of succeeding is 1 over 4 use numpy to find the probability at most one of Yelani and science succeed yeah there's it sounds weird but that's not the point so this is a point here so use numpy to find the probability Yelani succeeds but cyan does not succeed and finally divide the former by the latter probability I mean by doing this you're not just rephrasing the problem you're solving the problem if you know how to rephrase it this in this manner you already know how to solve the problem and you probably don't need codecs to do it for you so that's yeah I'm not sure how many of the solutions they have have such an intensive rephrasing and we'll see in a couple of seconds they've tried to kind of quantify the modifications they've done to the original problem statements but I'm afraid that the embeddings they've used can't capture this notion of how much you've actually helped solve the problem and it's gonna capture more of the yeah so I'm just gonna suspicious of how this similarity metric they've done is gonna capture this notion of actually solving the problem when rephrasing it if you know what I mean so finally we can see the the result here probability of Sam probability of Yelani blah blah just calculates and nicely names the variable so that's very cool things like it has comments it has a combination of informal and formal language here so you can see the the variable is very descriptive here and finally you can see that the vision which is not this thing is also a bit worrisome because I can see three dots here and yeah it'd be probably a bit cleaner if you assign this to a variable and then try to print it otherwise it's kind of weird anyhow you saw now the tweaks they've doing to make codecs work you can argue that some of them are are a bit like too helpful to codecs like the the number at the e problem we saw here but yeah let's see some other charts they have here but they've been additionally prompting the model with multiple problem statements and then sampling in other problem statement that codecs now invented and they've done a survey of those codecs generated like problem statements with the human generated problem statements and we can see the results here first off if we split on the x-axis you can see some MIT courses on the y-axis we can see difficulty and you can see that human written and machine generated problem statements are fairly close in difficulty according to this survey participants and then we have here like whether the the the problem statement was appropriate or not we can see that codecs generated problem statements are not as high quality as human ones but it still comes very very close and finally we have here a chart then that basically they've been given a problem statement and the question was hey is this generated by codecs by machine or by human and you can see that in the case when all of the problem statements were written by humans people sometimes thought that like like around 25% of those seemed like maybe machine generated those but when we're when they were presented with purely machine generated problem statements so here in this case people can kind of tell that something is not quite right but still they can judge and so half of the time they just say it's machines sometimes and then the other 50% of time they just say hey it's this is human generated okay let's see final charts here you can check out these ones yourself at your own pace I'm gonna just take a single one here is the one of the questions that the codecs generated it says find the area of the region bounded by the curve and the x-axis where the function is defined as y is equal to x squared times sine of x and the x-axis goes from 0 to pi and you can see the closest question in the data set looks very very similar but not quite there so find the area of the region under the given curve from 1 to 2 so they've defined as you can see X here is defined more formally here they've kind of used like an like human language to informal language to define the same thing and finally Y is defined similarly to codecs example here so maybe this one is very very close the only thing that codecs had to do is to change the function and change the the domain on the x-axis but like you can see some of these differ by quite a lot at least on the syntax level but looking at these generated questions I can I can kind of empathize with the participants in this survey here because it's really kind of tricky to tell which one of these is generated by a human and which one is generated by a by by codecs and that's that's fairly awesome finally finally this is the the similarity rate like metric I mentioned what I've done here is for every single course and again they have like 13 of those courses so here are the the MIT courses here is the Columbia course here are the courses from from that I guess the problems from that math benchmark and you can see that if this candle chart is close to one ideally that means that they didn't have to do any modification whatsoever so again stepping back for a second here what happened here is that they take the original problem statement they pass it to the I think Bert they take the modified problem statement and they do the same thing and then they just find the cosine similarity between those two like feature vectors and now the if they had minor modifications and those two vectors will be fairly similar and the cosine similarity will be close to one so what this basically shows you is that for example for this Columbia course they didn't have to to tweak the problems statements by a lot so that may either mean that that probably means that like those those problem statements are a bit easier for codecs to to to kind of parse and and generate the answer to you yeah you can see that for some of these they had to kind of do more tweaking so you can see here the the average cosine similarity is a bit more to the left compared to these other ones same here there is a huge variance here and the the average is quite on the left it's almost around 0.5 as I said it's kind of hard to looking at these quantitative results figure out so where I'm trying to get it here is that you can have the same cosine similarity for two modifications of the same original problem statement where one is a simple syntactic modification of the input problem statement whereas the other one is such rephrasing that you have actually solved the problem in a way so we saw that with the this problem this this example e so I'm trying to allude to this one here by by telling telling the codecs to divide the former by the letter probability they've kind of made it a lot easier so yeah it's kind of hard to understand through these visualizations how many times they had to do something like this if you yeah that's my point okay finally they have a similar thing just a histogram and you can see that all in all I think this is for all when you combine all the problem statements and then just accumulate and plot a histogram you can see that most of the time they had like 0.8 that's where the mode of this distribution lies basically yeah again it's hard to understand what this means in practice obviously be optimally if this mode was basically skewed all the way to the right here where we have cosine similarity of one which means there was no modification happening whatsoever and that's pretty much it I don't want to be too nitpicky here I mean the results are fairly astounding even with these tweaking I mean in two years time with better hardware bigger models more a novel like research ideas this can just get better cool hopefully like this video if you did consider subscribing click that notification bell share it with your friends and also join the discord community until next time bye
[{"start": 0.0, "end": 5.0, "text": " What's cracking guys? So as you can see here I finally got my camera back. I'm"}, {"start": 5.0, "end": 10.24, "text": " slowly setting up my YouTube studio here in London to its previous glory and yeah"}, {"start": 10.24, "end": 15.68, "text": " hopefully in the next couple weeks it's gonna get even better. Anyhow, so in"}, {"start": 15.68, "end": 20.52, "text": " this video I'm gonna cover this novel paper called A Neural Network Solves"}, {"start": 20.52, "end": 25.68, "text": " and Generates Mathematics Problems by Program Synthesis. Calculus, Differential"}, {"start": 25.68, "end": 32.04, "text": " Equations, Linear Algebra and more by this like it's a collaboration between a"}, {"start": 32.04, "end": 37.68, "text": " couple of universities among them MIT, Columbia, Harvard and University of"}, {"start": 37.68, "end": 43.739999999999995, "text": " Waterloo. So in a nutshell what I've done is this neural network here is a fancy"}, {"start": 43.739999999999995, "end": 49.760000000000005, "text": " way of saying codex from OpenAI and for those of you who are not familiar with"}, {"start": 49.760000000000005, "end": 53.8, "text": " the with the model I'm gonna don't worry I'm gonna cover it in a second give you"}, {"start": 53.8, "end": 56.959999999999994, "text": " some background some high-level understanding of what's going on. Hint"}, {"start": 56.959999999999994, "end": 64.64, "text": " hint it's just a GPT-3 model fine-tuned on codebase like on Python codebase and"}, {"start": 64.64, "end": 68.39999999999999, "text": " so what they've done what they've achieved is they've taken a bunch of"}, {"start": 68.39999999999999, "end": 72.96, "text": " problems from as you can see like these faculties such as MIT and they managed"}, {"start": 72.96, "end": 78.44, "text": " to solve every single problem which is amazing but the trick is you have to"}, {"start": 78.44, "end": 83.24, "text": " kind of tweak the input prompts you have to do the prompt programming magic to"}, {"start": 83.24, "end": 88.67999999999999, "text": " get this thing to work but in any case like it's very very very interesting"}, {"start": 88.67999999999999, "end": 93.56, "text": " results and yeah I'm gonna cover those in this in this video. So let me just"}, {"start": 93.56, "end": 98.72, "text": " show you what is going on how the how the pipeline looks like basically"}, {"start": 98.72, "end": 104.11999999999999, "text": " basically you have something like this so here is an example of problem A and"}, {"start": 104.11999999999999, "end": 110.39999999999999, "text": " so the question is find the differential oops let me just one no doesn't want to"}, {"start": 110.4, "end": 116.32000000000001, "text": " render here okay so find the differential of W where W equals like"}, {"start": 116.32000000000001, "end": 122.88000000000001, "text": " a natural like logarithm of X square plus Y squared plus Z squared and"}, {"start": 122.88000000000001, "end": 130.44, "text": " basically want to find the derivatives with respect to both X Y and Z and you"}, {"start": 130.44, "end": 135.52, "text": " can see what it do here they take that that prompt and they basically rephrase"}, {"start": 135.52, "end": 139.06, "text": " it in a different way here so in differential equations they add some"}, {"start": 139.06, "end": 145.28, "text": " additional context and then they feed that into codecs which you sample from"}, {"start": 145.28, "end": 150.36, "text": " to generate this snippet of Python text and I deliberately say text because this"}, {"start": 150.36, "end": 155.2, "text": " is nothing than a large language model pre-chained on code which is just"}, {"start": 155.2, "end": 163.76, "text": " a like a specific type of text so then they run the code and basically they"}, {"start": 163.76, "end": 168.68, "text": " show that every single unit test passed for all of the they actually show that"}, {"start": 168.68, "end": 174.92000000000002, "text": " all the solutions that codecs generated were were correct with minor or bigger"}, {"start": 174.92000000000002, "end": 179.92000000000002, "text": " tweaks as we're going to see a bit later but that's it in a nutshell so if you're"}, {"start": 179.92000000000002, "end": 185.36, "text": " not familiar with codecs let me now briefly briefly show you what codecs is"}, {"start": 185.36, "end": 190.60000000000002, "text": " all about so evaluating large language models trained on code so this is the"}, {"start": 190.60000000000002, "end": 197.12, "text": " the codecs paper and you can see a group of people from OpenAI I guess a large"}, {"start": 197.12, "end": 203.08, "text": " proportion of their team and you shouldn't be surprised when you when you"}, {"start": 203.08, "end": 207.4, "text": " see when you see a bunch of authors on a paper it basically means that this was"}, {"start": 207.4, "end": 212.64000000000001, "text": " more of a like a engineering effort rather than trying to invent something"}, {"start": 212.64000000000001, "end": 222.36, "text": " some new machine learning like idea or whatnot so what's going on here one"}, {"start": 222.36, "end": 225.96, "text": " thing too worth mentioning is you you probably heard of github copilot"}, {"start": 225.96, "end": 232.4, "text": " basically it's it's powered whoops I crossed that oh my god so github copilot"}, {"start": 232.4, "end": 238.60000000000002, "text": " is basically just using codecs improved version of the codecs from this paper in"}, {"start": 238.60000000000002, "end": 243.84, "text": " the background so yeah maybe that's a that's like a kind of fun fact but like"}, {"start": 243.84, "end": 248.8, "text": " let me jump and explain how this thing works so as I said it's simply a GPT-3"}, {"start": 248.8, "end": 255.34, "text": " model fine-tuned so pre-trained GPT-3 model fine-tuned on this on this code"}, {"start": 255.34, "end": 261.82, "text": " base on this data set that was collected by scraping various Python files from"}, {"start": 261.82, "end": 267.22, "text": " github and then post hoc doing some filtering so now the important detail"}, {"start": 267.22, "end": 271.8, "text": " here is that you don't actually have to have a pre-trained GPT-3 model that's"}, {"start": 271.8, "end": 278.0, "text": " what I showed okay let's see that part here basically they say that surprisingly"}, {"start": 278.0, "end": 282.82, "text": " we did not observe improvements when starting from a pre-trained language"}, {"start": 282.82, "end": 288.76, "text": " model possibly because the fine-tuned fine-tuning data set is so large so by"}, {"start": 288.76, "end": 296.38, "text": " so large what they actually mean is 159 gigabytes of text which they got after"}, {"start": 296.38, "end": 301.24, "text": " collecting as I said from github and then doing some post hoc filtering to"}, {"start": 301.24, "end": 307.0, "text": " get this data set here so nevertheless models fine-tuned from GPT converge more"}, {"start": 307.0, "end": 312.44, "text": " quickly so we apply this strategy for all subsequent experiments so basically"}, {"start": 312.44, "end": 318.88, "text": " like the pre-training part is not about the same thing as if you've read the"}, {"start": 318.88, "end": 322.71999999999997, "text": " Universal Transformers or Universal Computational Engines I think that's the"}, {"start": 322.71999999999997, "end": 328.84, "text": " name of the of the of the paper so the idea is not to use the knowledge like"}, {"start": 328.84, "end": 333.15999999999997, "text": " the idea is not to leverage the knowledge the the problem-solving logic"}, {"start": 333.15999999999997, "end": 338.36, "text": " that GPT-3 obtained by by pre-training on this huge corpus of text the idea is"}, {"start": 338.36, "end": 341.32, "text": " just to bootstrap it to make it a bit quicker so this is more of an"}, {"start": 341.32, "end": 346.96, "text": " optimization detail I guess than the actual important detail"}, {"start": 346.96, "end": 354.24, "text": " so if you remember the Universal Computational Engine paper the idea was"}, {"start": 354.24, "end": 359.84, "text": " that transformers when you pre-train them using the usual unsupervised like"}, {"start": 359.84, "end": 365.12, "text": " pre-training objective so you've got your transformer here and the idea is"}, {"start": 365.12, "end": 370.12, "text": " that the weights that you obtained by doing that by pre-training it on a huge"}, {"start": 370.12, "end": 375.48, "text": " corpus of text is those weights are kind of very generic and they can solve"}, {"start": 375.48, "end": 380.16, "text": " problems for like whatever modality you want so what they've showed is that you"}, {"start": 380.16, "end": 387.96, "text": " can kind of take the input layer the embeddings and discard the old ones as"}, {"start": 387.96, "end": 391.48, "text": " well as the like whatever like classification heads in the case of"}, {"start": 391.48, "end": 396.52, "text": " classification problems just kind of fine-tune these parts and you can apply"}, {"start": 396.52, "end": 402.52, "text": " then the same transformer to like images or whatnot so anyhow in this"}, {"start": 402.52, "end": 406.79999999999995, "text": " case in this paper they are not actually using to fine-tune they are more they're"}, {"start": 406.79999999999995, "end": 412.79999999999995, "text": " using the the pre-trained weights to bootstrap the training process anyhow"}, {"start": 412.79999999999995, "end": 419.76, "text": " let's get back to the explanation here so what you do is you prompt the codex"}, {"start": 419.76, "end": 424.24, "text": " with this thing here so this is your prompt and then you basically"}, {"start": 424.24, "end": 428.28000000000003, "text": " generate this piece of text the same thing as with GPT-3 which in this case"}, {"start": 428.28000000000003, "end": 433.32, "text": " is code so and this code probably passes the unit test for this particular"}, {"start": 433.32, "end": 439.0, "text": " function usually the context is such that you have that you have the the"}, {"start": 439.0, "end": 443.76, "text": " signature here so you have the signature of the function right here then you have"}, {"start": 443.76, "end": 449.8, "text": " this part the the the the doc string and then you prompt and create the the body"}, {"start": 449.8, "end": 453.6, "text": " part of the function and that's how people usually use github copilot as"}, {"start": 453.6, "end": 458.96000000000004, "text": " well so you may be familiar with that format so let's see what what the codex"}, {"start": 458.96000000000004, "end": 464.1, "text": " brought onto the table so again codex is just a GPT-3 model pre-trained on a"}, {"start": 464.1, "end": 469.26000000000005, "text": " particular type of text which is code so we are basically just narrowing the"}, {"start": 469.26000000000005, "end": 475.12, "text": " distribution of the data that we want codex to learn how to model and the"}, {"start": 475.12, "end": 480.72, "text": " second innovation here is innovation I mean this is not a novel idea but it's a"}, {"start": 480.72, "end": 486.40000000000003, "text": " very neat idea to use unit tests to make sure that your your model is working as"}, {"start": 486.40000000000003, "end": 492.32000000000005, "text": " expected and because of unit tests what they can afford to do is generate like"}, {"start": 492.32000000000005, "end": 499.76000000000005, "text": " case samples for example hundred samples and see whether any of those passes all"}, {"start": 499.76000000000005, "end": 504.36, "text": " of the unit tests for that particular function we're currently trying to to to"}, {"start": 504.36, "end": 510.6, "text": " create to generate and they introduce this this metric called I think it's called"}, {"start": 510.6, "end": 519.16, "text": " pass at K where K is the number of samples they generate so it's usually I"}, {"start": 519.16, "end": 526.84, "text": " think they usually use either one or or hundred etc and basically they treat it"}, {"start": 526.84, "end": 530.64, "text": " as a pass if any one of those samples passes all of the unit tests so you can"}, {"start": 530.64, "end": 537.16, "text": " imagine if you had like 50 examples in the data set and so 50 and for all of"}, {"start": 537.16, "end": 542.28, "text": " those we the one some of these samples managed to pass all of the unit tests"}, {"start": 542.28, "end": 548.64, "text": " then this pass at K metric would be equal to 1 and subsequently if you if you"}, {"start": 548.64, "end": 553.88, "text": " if some of the functions if none of the samples managed to produce to pass all"}, {"start": 553.88, "end": 557.84, "text": " the unit tests and obviously pass at K is gonna decrease towards towards zero"}, {"start": 557.84, "end": 563.72, "text": " hopefully that's very very self-explanatory but the thing is you"}, {"start": 563.72, "end": 568.88, "text": " don't have the luxury of generating hundred samples when somebody is trying"}, {"start": 568.88, "end": 575.64, "text": " to use codecs to for example generate novel code because you don't have usually"}, {"start": 575.64, "end": 579.1600000000001, "text": " usually don't have like unit tests unless you're doing test-driven"}, {"start": 579.1600000000001, "end": 584.36, "text": " development which people are usually not doing and so what they had to figure"}, {"start": 584.36, "end": 590.28, "text": " out is how to generate the best possible sample from codecs and they just devised"}, {"start": 590.28, "end": 595.5600000000001, "text": " this mean logprop heuristic which is fairly simple and they tried a couple"}, {"start": 595.5600000000001, "end": 600.48, "text": " other things like they tried instead of using mean they tried using some and it"}, {"start": 600.48, "end": 605.32, "text": " just this thing just worked worked the best so the idea is the following let me"}, {"start": 605.32, "end": 612.4, "text": " try and kind of draw it down here so you have your your your codecs model here"}, {"start": 612.4, "end": 619.68, "text": " and you prompt it with some with some prompt as we saw on the on the chart"}, {"start": 619.68, "end": 625.56, "text": " there so we prompt it with something here and then we start generating okay"}, {"start": 625.56, "end": 632.36, "text": " so you generate the first token you and how you generate is you basically have"}, {"start": 632.36, "end": 635.88, "text": " to sample from a distribution over tokens so let's say we had some nice"}, {"start": 635.88, "end": 641.3199999999999, "text": " distribution like this so basically if you're using greedy you'd be you'd be"}, {"start": 641.32, "end": 645.4000000000001, "text": " sampling exactly this token so let me change the color so you'd sample this"}, {"start": 645.4000000000001, "end": 651.36, "text": " token here so whatever is this token here so you take that token oops and you"}, {"start": 651.36, "end": 657.2800000000001, "text": " then feed that token here you'd embed the token obviously so you pass the"}, {"start": 657.2800000000001, "end": 661.84, "text": " vector in and then you get another output here and then you just repeat"}, {"start": 661.84, "end": 666.96, "text": " that and so what what the what the mean logprop heuristic does is for all of"}, {"start": 666.96, "end": 671.84, "text": " degenerated tokens you just take their mean so you just so for all the tokens"}, {"start": 671.84, "end": 677.84, "text": " you just take this this probability and log log of the probability and you just"}, {"start": 677.84, "end": 681.38, "text": " do the mean so intuitively what the heuristic is trying to do is capture"}, {"start": 681.38, "end": 686.4000000000001, "text": " those samples that are kind of very probable under under this codecs model"}, {"start": 686.4000000000001, "end": 691.5600000000001, "text": " and that's that's the whole idea and that's pretty much it so additionally"}, {"start": 691.56, "end": 697.4799999999999, "text": " they had some part where they find you know even like on data sets which are"}, {"start": 697.4799999999999, "end": 701.4, "text": " more pure than just simply collecting data from from github so let me find"}, {"start": 701.4, "end": 705.7199999999999, "text": " that thing somewhere here so supervised fine-tuning is a section of the paper"}, {"start": 705.7199999999999, "end": 712.76, "text": " and what I've done here is they've they've taken only the code from"}, {"start": 712.76, "end": 717.8, "text": " competitive programming websites which is usually of higher quality than if you"}, {"start": 717.8, "end": 722.24, "text": " were to just randomly scrape the code from the internet from the from github"}, {"start": 722.24, "end": 727.56, "text": " in particular I think and so they collect the data by as I said competitive"}, {"start": 727.56, "end": 732.56, "text": " programming websites plus they found those repos which have continuous"}, {"start": 732.56, "end": 736.9, "text": " integration in place and they kind of found the logic to extract the functions"}, {"start": 736.9, "end": 741.4, "text": " and use those functions as training examples and by doing this they created"}, {"start": 741.4, "end": 746.8399999999999, "text": " codecs as family of models and those perform even better looking at the pass"}, {"start": 746.84, "end": 752.0, "text": " at K metric compared to the regular codecs so you can imagine that github"}, {"start": 752.0, "end": 757.5600000000001, "text": " copilot is using some combination of these techniques let's see how this"}, {"start": 757.5600000000001, "end": 766.36, "text": " novel paper leveraged codecs to solve these university level problems they use"}, {"start": 766.36, "end": 772.08, "text": " the problems as I mentioned from MIT various MIT courses from this Columbia"}, {"start": 772.08, "end": 777.0, "text": " course a computational linear algebra and there is this additional benchmark"}, {"start": 777.0, "end": 781.24, "text": " called math topics and they just do problems from there so what they do is"}, {"start": 781.24, "end": 785.84, "text": " they take a particular problem to be more precise they're using they take a"}, {"start": 785.84, "end": 791.76, "text": " question for that mathematical problem and they do three things so they either"}, {"start": 791.76, "end": 798.72, "text": " they either tidy questions so simplify a word problem to mathematical content or"}, {"start": 798.72, "end": 805.08, "text": " they add some context like topic library a name for example they hint to to codecs"}, {"start": 805.08, "end": 811.1600000000001, "text": " hey use numpy or hey use matplotlib and they sometimes define the concepts"}, {"start": 811.1600000000001, "end": 816.1600000000001, "text": " because it's expected that students who are attending the course already have"}, {"start": 816.1600000000001, "end": 820.6, "text": " that context and so it's kind of unfair not to give the same context to codecs so"}, {"start": 820.6, "end": 825.52, "text": " they provide a context and finally sometimes they have to interactively"}, {"start": 825.52, "end": 831.1999999999999, "text": " refine the prompt on until they they solve the problem so yeah you can see"}, {"start": 831.1999999999999, "end": 835.4399999999999, "text": " some problems here it's one thing when you know the actual solution that's"}, {"start": 835.4399999999999, "end": 839.1999999999999, "text": " that's cool but like when you don't know the solution this obviously will not"}, {"start": 839.1999999999999, "end": 845.16, "text": " work so it's very the application area is actually universities and university"}, {"start": 845.16, "end": 850.56, "text": " classes so it's got a huge potential there and here are some examples of what"}, {"start": 850.56, "end": 856.4399999999999, "text": " codecs managed to do so codecs generated a snippet of code of Python code in"}, {"start": 856.4399999999999, "end": 862.7199999999999, "text": " particular in this paper at least and the code generated these plots and solve"}, {"start": 862.7199999999999, "end": 869.04, "text": " this problem actually so so the problem was to like what's the volume you get if"}, {"start": 869.04, "end": 874.0799999999999, "text": " you rotate this 2d chart around the third axis so what what happens if you"}, {"start": 874.0799999999999, "end": 880.1999999999999, "text": " have for example imagine this is 3d imagine you add an axis here and you try"}, {"start": 880.2, "end": 885.44, "text": " and rotate this this 2d chart around this axis and you eventually get this"}, {"start": 885.44, "end": 889.76, "text": " 3d body here so that's what codecs had to kind of figure out and in this"}, {"start": 889.76, "end": 893.2, "text": " particular question and there are many other things that that it managed to"}, {"start": 893.2, "end": 897.9200000000001, "text": " solve obviously every single question they throw at codecs it managed it"}, {"start": 897.9200000000001, "end": 903.32, "text": " managed to solve it after some some prompt hacking as I like to call these"}, {"start": 903.32, "end": 909.6400000000001, "text": " things here so let's see what is going on here so one one thing you may think"}, {"start": 909.64, "end": 914.88, "text": " here is that what if codecs during the the pre training procedure which took"}, {"start": 914.88, "end": 920.56, "text": " into account the whole like a huge amount of code on the internet among"}, {"start": 920.56, "end": 925.0, "text": " those you can be certain that some of the code is used to solve some of the"}, {"start": 925.0, "end": 929.38, "text": " mathematical problems from these specific universities like MIT etc so"}, {"start": 929.38, "end": 934.68, "text": " that was my concern for sure and they did a good job there obviously by by"}, {"start": 934.68, "end": 939.2, "text": " yeah they sit here we validate that our results are not merely overfitting"}, {"start": 939.2, "end": 943.6400000000001, "text": " training data by solving a new computational linear algebra course"}, {"start": 943.6400000000001, "end": 948.76, "text": " which is not available online and is therefore unseen during codecs training"}, {"start": 948.76, "end": 955.88, "text": " we obtain correct answer for all of the randomly sampled questions and that's"}, {"start": 955.88, "end": 963.08, "text": " that's kind of a nice result okay so that out of the way let's continue here"}, {"start": 963.08, "end": 969.0, "text": " here you can see some some example questions from from these courses so"}, {"start": 969.0, "end": 975.0, "text": " they have 13 courses in in in total and for each one of those they have a"}, {"start": 975.0, "end": 980.32, "text": " representative example here and you can see for example one of those in in the"}, {"start": 980.32, "end": 985.08, "text": " multivariable calculus course is describe the graph of the function F and"}, {"start": 985.08, "end": 991.0, "text": " F is a multivariable function of X and Y equals to 10 minus as you can see here"}, {"start": 991.0, "end": 996.48, "text": " and what codecs managed to do is probably use modplotlib and basically"}, {"start": 996.48, "end": 1001.2, "text": " plotted the function in 3d as you can see here and yeah you can go at your"}, {"start": 1001.2, "end": 1005.52, "text": " own pace through these problems if you're if you're curious but like they"}, {"start": 1005.52, "end": 1012.4, "text": " vary a lot and in the sense that sometimes you have to plot the the"}, {"start": 1012.4, "end": 1016.72, "text": " function sometimes you just have to give a numerical answer like let me find some"}, {"start": 1016.72, "end": 1024.76, "text": " some some simple example what is the greatest common factor of 84 112 and 210"}, {"start": 1024.76, "end": 1033.56, "text": " and the model outputs 14 which is great okay so that's it now let me dig deeper"}, {"start": 1033.56, "end": 1040.2, "text": " into what exactly do they do how do they hack that the prompts so that the codecs"}, {"start": 1040.2, "end": 1047.3600000000001, "text": " generates correct outputs so first things first is here they added the"}, {"start": 1047.3600000000001, "end": 1053.8, "text": " in differential equations so that's providing the topic context and then"}, {"start": 1053.8, "end": 1057.48, "text": " they kind of rearrange this so you can see it's a it's a mix of multiple things"}, {"start": 1057.48, "end": 1062.32, "text": " so here in particular they not only provide the topic context they also"}, {"start": 1062.32, "end": 1068.96, "text": " provide the library context you can see here simpy is is a library and they kind"}, {"start": 1068.96, "end": 1072.8400000000001, "text": " of rearrange the question and this is the code that that that that codecs"}, {"start": 1072.8400000000001, "end": 1078.1200000000001, "text": " generates so it imports the simpy correctly it then generates the the these"}, {"start": 1078.1200000000001, "end": 1084.44, "text": " abstract symbols here x y and z and then just does the log as the function here"}, {"start": 1084.44, "end": 1092.72, "text": " is telling it and finally it just prints that if like derivatives of W of W with"}, {"start": 1092.72, "end": 1097.58, "text": " respect to X Y and Z and this is one of the simpler examples actually you can"}, {"start": 1097.58, "end": 1101.6799999999998, "text": " you can later there is a huge appendix and they codecs sometimes generates"}, {"start": 1101.6799999999998, "end": 1109.36, "text": " fairly fairly complex programs here is another another example here they they"}, {"start": 1109.36, "end": 1114.6399999999999, "text": " add multiple library hints they have some question here and they rephrase it"}, {"start": 1114.6399999999999, "end": 1119.78, "text": " and they put simpy they put stream stream plot and you can indeed see that"}, {"start": 1119.78, "end": 1129.36, "text": " stream plot is used here as well as the simpy so that's cool finally here is the"}, {"start": 1129.36, "end": 1133.84, "text": " context in the form of definitions okay so let's see what I've done here"}, {"start": 1133.84, "end": 1139.36, "text": " calculate the probability of getting a two-pair poker hand and use a human if"}, {"start": 1139.36, "end": 1143.2, "text": " you're not familiar with poker rules will have difficulty solving this"}, {"start": 1143.2, "end": 1146.8, "text": " because you don't know what these things are in the first place so that's why they"}, {"start": 1146.8, "end": 1151.8, "text": " sometimes have to add additional context in the form of definitions so a hand is"}, {"start": 1151.8, "end": 1157.36, "text": " a set of five cards that are drawn randomly from a standard 52 card deck"}, {"start": 1157.36, "end": 1163.6599999999999, "text": " with 13 ranks of four cards each etc etc and finally they hint them out here so"}, {"start": 1163.6599999999999, "end": 1169.24, "text": " write a program that generates simulations so it's additionally telling"}, {"start": 1169.24, "end": 1175.96, "text": " the codecs to generate the program and like a four in a form of a simulation so"}, {"start": 1175.96, "end": 1179.48, "text": " all of these help the model generate the correct solution and you can see here"}, {"start": 1179.48, "end": 1186.88, "text": " fairly fairly nice nice like a for loop it does a simulation where it does like"}, {"start": 1186.88, "end": 1192.92, "text": " what's this like 10 million iterations 10 million I guess iterations in this"}, {"start": 1192.92, "end": 1198.16, "text": " simulation at shuffles the deck takes five cards first five cards so it's like"}, {"start": 1198.16, "end": 1203.4, "text": " a not that optimal way to do this but like still it's it's it's semantically"}, {"start": 1203.4, "end": 1209.24, "text": " sound and it takes the first five which is a random hand and then it checks"}, {"start": 1209.24, "end": 1214.0, "text": " whether it has two pairs if the hand has two pairs then we just increment and"}, {"start": 1214.0, "end": 1219.8400000000001, "text": " finally it returns the diffraction so how likely is it to have like a two pair"}, {"start": 1219.8400000000001, "end": 1223.88, "text": " if you just draw random hands as it was asked here so calculate the probability"}, {"start": 1223.88, "end": 1228.2, "text": " of getting a two pair poker hand and the model did exactly that so that's cool"}, {"start": 1228.2, "end": 1233.0800000000002, "text": " so these examples now get a bit more problematic so you can see here that"}, {"start": 1233.08, "end": 1238.28, "text": " this is way more intricate so what it done is they've split the problem a"}, {"start": 1238.28, "end": 1242.6, "text": " statement into multiple parts so they first write like they first say here"}, {"start": 1242.6, "end": 1248.1599999999999, "text": " write a program that draws the Lorenz strange attractor and you can see that"}, {"start": 1248.1599999999999, "end": 1253.12, "text": " codecs generate some code and we have some some some some results here now"}, {"start": 1253.12, "end": 1257.62, "text": " here I'm not kind of sure whether they're just appending this to the"}, {"start": 1257.62, "end": 1264.04, "text": " previous prompt including this output so I'm not sure whether they just take this"}, {"start": 1264.04, "end": 1269.36, "text": " and then concatenate that with this snippet here and then they just add this"}, {"start": 1269.36, "end": 1276.6, "text": " in order to get this or do they just kind of yeah I guess that makes a lot of"}, {"start": 1276.6, "end": 1282.12, "text": " sense although that can very soon become unwieldy if the output is very very"}, {"start": 1282.12, "end": 1287.32, "text": " large because transformers as you recall have memory issues you can't"}, {"start": 1287.32, "end": 1293.28, "text": " have infinitely long sequences so yeah you can see they had to do produce the"}, {"start": 1293.28, "end": 1297.12, "text": " X projection of the Lawrence trajectory and then outcomes in our snippet and"}, {"start": 1297.12, "end": 1300.6799999999998, "text": " then plot the XY projection of the Lawrence trajectory and then comes in"}, {"start": 1300.6799999999998, "end": 1306.9199999999998, "text": " another snippet and finally here's our last example here and let me read it to"}, {"start": 1306.9199999999998, "end": 1313.06, "text": " you outside of their humdrum duties as 6042 TAs teaching assistants cyan is"}, {"start": 1313.06, "end": 1317.44, "text": " trying to learn to levitate using only intense concentration and Yelani is"}, {"start": 1317.44, "end": 1323.8, "text": " launching a Nelson 2008 presidential campaign suppose that science probability"}, {"start": 1323.8, "end": 1329.56, "text": " of levitating is 1 over 6 Yelani's chance of becoming president is 1 over 4"}, {"start": 1329.56, "end": 1336.0, "text": " and the success of one does not alter the others chances if at most one of"}, {"start": 1336.0, "end": 1340.24, "text": " them succeeds what is the probability that Yelani becomes the president of the"}, {"start": 1340.24, "end": 1346.48, "text": " United States this is a fairly complex problem statement and the things that"}, {"start": 1346.48, "end": 1351.8, "text": " kind of bothers me here is that they've actually doing this rephrasing procedure"}, {"start": 1351.8, "end": 1356.32, "text": " they've kind of solved and made this problem easier now you can see for"}, {"start": 1356.32, "end": 1360.56, "text": " yourself whether that's true or not but like they say here that suppose that"}, {"start": 1360.56, "end": 1365.28, "text": " science probability science succeeding is 1 over 6 I guess it should be off"}, {"start": 1365.28, "end": 1371.0, "text": " science succeeding here or something like that and Yelani's probability of succeeding is 1"}, {"start": 1371.0, "end": 1377.04, "text": " over 4 use numpy to find the probability at most one of Yelani and"}, {"start": 1377.04, "end": 1382.16, "text": " science succeed yeah there's it sounds weird but that's not the point so this"}, {"start": 1382.16, "end": 1385.84, "text": " is a point here so use numpy to find the probability Yelani succeeds but"}, {"start": 1385.84, "end": 1390.56, "text": " cyan does not succeed and finally divide the former by the latter probability I"}, {"start": 1390.56, "end": 1395.6599999999999, "text": " mean by doing this you're not just rephrasing the problem you're solving the"}, {"start": 1395.6599999999999, "end": 1401.6399999999999, "text": " problem if you know how to rephrase it this in this manner you already know how"}, {"start": 1401.6399999999999, "end": 1405.04, "text": " to solve the problem and you probably don't need codecs to do it for you so"}, {"start": 1405.04, "end": 1410.8, "text": " that's yeah I'm not sure how many of the solutions they have have such an"}, {"start": 1410.8, "end": 1415.0, "text": " intensive rephrasing and we'll see in a couple of seconds they've tried to kind"}, {"start": 1415.0, "end": 1420.92, "text": " of quantify the modifications they've done to the original problem statements"}, {"start": 1420.92, "end": 1425.96, "text": " but I'm afraid that the embeddings they've used can't capture this notion"}, {"start": 1425.96, "end": 1431.24, "text": " of how much you've actually helped solve the problem and it's gonna capture more"}, {"start": 1431.24, "end": 1440.96, "text": " of the yeah so I'm just gonna suspicious of how this similarity metric they've"}, {"start": 1440.96, "end": 1445.52, "text": " done is gonna capture this notion of actually solving the problem when"}, {"start": 1445.52, "end": 1450.44, "text": " rephrasing it if you know what I mean so finally we can see the the result here"}, {"start": 1450.44, "end": 1456.8400000000001, "text": " probability of Sam probability of Yelani blah blah just calculates and nicely"}, {"start": 1456.8400000000001, "end": 1461.64, "text": " names the variable so that's very cool things like it has comments it has a"}, {"start": 1461.64, "end": 1465.68, "text": " combination of informal and formal language here so you can see the the"}, {"start": 1465.68, "end": 1469.24, "text": " variable is very descriptive here and finally you can see that the vision"}, {"start": 1469.24, "end": 1474.96, "text": " which is not this thing is also a bit worrisome because I can see three dots"}, {"start": 1474.96, "end": 1481.44, "text": " here and yeah it'd be probably a bit cleaner if you assign this to a variable"}, {"start": 1481.44, "end": 1488.26, "text": " and then try to print it otherwise it's kind of weird anyhow you saw now the"}, {"start": 1488.26, "end": 1493.64, "text": " tweaks they've doing to make codecs work you can argue that some of them are"}, {"start": 1493.64, "end": 1501.4, "text": " are a bit like too helpful to codecs like the the number at the e problem we saw"}, {"start": 1501.4, "end": 1506.8000000000002, "text": " here but yeah let's see some other charts they have here but they've been"}, {"start": 1506.8000000000002, "end": 1512.6000000000001, "text": " additionally prompting the model with multiple problem statements and then"}, {"start": 1512.6000000000001, "end": 1519.72, "text": " sampling in other problem statement that codecs now invented and they've done a"}, {"start": 1519.72, "end": 1526.16, "text": " survey of those codecs generated like problem statements with the human"}, {"start": 1526.16, "end": 1531.66, "text": " generated problem statements and we can see the results here first off if we"}, {"start": 1531.66, "end": 1536.8, "text": " split on the x-axis you can see some MIT courses on the y-axis we can see"}, {"start": 1536.8, "end": 1541.8600000000001, "text": " difficulty and you can see that human written and machine generated problem"}, {"start": 1541.8600000000001, "end": 1548.08, "text": " statements are fairly close in difficulty according to this survey"}, {"start": 1548.08, "end": 1557.8, "text": " participants and then we have here like whether the the the problem statement"}, {"start": 1557.8, "end": 1562.76, "text": " was appropriate or not we can see that codecs generated problem statements are"}, {"start": 1562.76, "end": 1568.4399999999998, "text": " not as high quality as human ones but it still comes very very close and finally"}, {"start": 1568.4399999999998, "end": 1574.4399999999998, "text": " we have here a chart then that basically they've been given a problem statement"}, {"start": 1574.44, "end": 1579.44, "text": " and the question was hey is this generated by codecs by machine or by"}, {"start": 1579.44, "end": 1584.72, "text": " human and you can see that in the case when all of the problem statements were"}, {"start": 1584.72, "end": 1591.6000000000001, "text": " written by humans people sometimes thought that like like around 25% of"}, {"start": 1591.6000000000001, "end": 1595.88, "text": " those seemed like maybe machine generated those but when we're when they"}, {"start": 1595.88, "end": 1599.52, "text": " were presented with purely machine generated problem statements so here in"}, {"start": 1599.52, "end": 1603.44, "text": " this case people can kind of tell that something is not quite right but still"}, {"start": 1603.44, "end": 1608.72, "text": " they can judge and so half of the time they just say it's machines sometimes"}, {"start": 1608.72, "end": 1614.64, "text": " and then the other 50% of time they just say hey it's this is human generated"}, {"start": 1614.64, "end": 1623.64, "text": " okay let's see final charts here you can check out these ones yourself at your"}, {"start": 1623.64, "end": 1627.96, "text": " own pace I'm gonna just take a single one here is the one of the questions"}, {"start": 1627.96, "end": 1634.44, "text": " that the codecs generated it says find the area of the region bounded by the"}, {"start": 1634.44, "end": 1642.2, "text": " curve and the x-axis where the function is defined as y is equal to x squared"}, {"start": 1642.2, "end": 1649.64, "text": " times sine of x and the x-axis goes from 0 to pi and you can see the closest"}, {"start": 1649.64, "end": 1655.08, "text": " question in the data set looks very very similar but not quite there so find the"}, {"start": 1655.08, "end": 1660.56, "text": " area of the region under the given curve from 1 to 2 so they've defined as you"}, {"start": 1660.56, "end": 1667.56, "text": " can see X here is defined more formally here they've kind of used like an like"}, {"start": 1667.56, "end": 1673.72, "text": " human language to informal language to define the same thing and finally Y is"}, {"start": 1673.72, "end": 1679.86, "text": " defined similarly to codecs example here so maybe this one is very very"}, {"start": 1679.86, "end": 1685.24, "text": " close the only thing that codecs had to do is to change the function and change"}, {"start": 1685.24, "end": 1689.9599999999998, "text": " the the domain on the x-axis but like you can see some of these differ by"}, {"start": 1689.9599999999998, "end": 1694.9599999999998, "text": " quite a lot at least on the syntax level but looking at these generated"}, {"start": 1694.9599999999998, "end": 1700.24, "text": " questions I can I can kind of empathize with the participants in this survey"}, {"start": 1700.24, "end": 1704.4599999999998, "text": " here because it's really kind of tricky to tell which one of these is generated"}, {"start": 1704.4599999999998, "end": 1708.6399999999999, "text": " by a human and which one is generated by a by by codecs and that's that's fairly"}, {"start": 1708.64, "end": 1715.64, "text": " awesome finally finally this is the the similarity rate like metric I mentioned"}, {"start": 1715.64, "end": 1721.3200000000002, "text": " what I've done here is for every single course and again they have like 13 of"}, {"start": 1721.3200000000002, "end": 1726.68, "text": " those courses so here are the the MIT courses here is the Columbia course here"}, {"start": 1726.68, "end": 1732.16, "text": " are the courses from from that I guess the problems from that math benchmark"}, {"start": 1732.16, "end": 1741.28, "text": " and you can see that if this candle chart is close to one ideally that means"}, {"start": 1741.28, "end": 1746.68, "text": " that they didn't have to do any modification whatsoever so again"}, {"start": 1746.68, "end": 1750.92, "text": " stepping back for a second here what happened here is that they take the"}, {"start": 1750.92, "end": 1756.2, "text": " original problem statement they pass it to the I think Bert they take the"}, {"start": 1756.2, "end": 1760.8400000000001, "text": " modified problem statement and they do the same thing and then they just find"}, {"start": 1760.84, "end": 1767.9199999999998, "text": " the cosine similarity between those two like feature vectors and now the if they"}, {"start": 1767.9199999999998, "end": 1771.8799999999999, "text": " had minor modifications and those two vectors will be fairly similar and the"}, {"start": 1771.8799999999999, "end": 1777.28, "text": " cosine similarity will be close to one so what this basically shows you is that"}, {"start": 1777.28, "end": 1783.12, "text": " for example for this Columbia course they didn't have to to tweak the problems"}, {"start": 1783.12, "end": 1788.0, "text": " statements by a lot so that may either mean that that probably means that like"}, {"start": 1788.0, "end": 1792.16, "text": " those those problem statements are a bit easier for codecs to to to kind of parse"}, {"start": 1792.16, "end": 1798.92, "text": " and and generate the answer to you yeah you can see that for some of these they"}, {"start": 1798.92, "end": 1804.8, "text": " had to kind of do more tweaking so you can see here the the average cosine"}, {"start": 1804.8, "end": 1810.56, "text": " similarity is a bit more to the left compared to these other ones same here"}, {"start": 1810.56, "end": 1816.24, "text": " there is a huge variance here and the the average is quite on the left it's"}, {"start": 1816.24, "end": 1821.32, "text": " almost around 0.5 as I said it's kind of hard to looking at these quantitative"}, {"start": 1821.32, "end": 1826.56, "text": " results figure out so where I'm trying to get it here is that you can have the"}, {"start": 1826.56, "end": 1831.08, "text": " same cosine similarity for two modifications of the same original"}, {"start": 1831.08, "end": 1837.08, "text": " problem statement where one is a simple syntactic modification of the input"}, {"start": 1837.08, "end": 1841.88, "text": " problem statement whereas the other one is such rephrasing that you have actually"}, {"start": 1841.88, "end": 1848.88, "text": " solved the problem in a way so we saw that with the this problem this this"}, {"start": 1848.88, "end": 1855.68, "text": " example e so I'm trying to allude to this one here by by telling telling the"}, {"start": 1855.68, "end": 1859.2, "text": " codecs to divide the former by the letter probability they've kind of made"}, {"start": 1859.2, "end": 1862.88, "text": " it a lot easier so yeah it's kind of hard to understand through these"}, {"start": 1862.88, "end": 1867.44, "text": " visualizations how many times they had to do something like this if you yeah"}, {"start": 1867.44, "end": 1874.0800000000002, "text": " that's my point okay finally they have a similar thing just a histogram and you"}, {"start": 1874.0800000000002, "end": 1878.42, "text": " can see that all in all I think this is for all when you combine all the problem"}, {"start": 1878.42, "end": 1882.96, "text": " statements and then just accumulate and plot a histogram you can see that most"}, {"start": 1882.96, "end": 1889.16, "text": " of the time they had like 0.8 that's where the mode of this distribution lies"}, {"start": 1889.16, "end": 1894.68, "text": " basically yeah again it's hard to understand what this means in practice"}, {"start": 1894.68, "end": 1902.4, "text": " obviously be optimally if this mode was basically skewed all the way to the"}, {"start": 1902.4, "end": 1907.64, "text": " right here where we have cosine similarity of one which means there was"}, {"start": 1907.64, "end": 1913.04, "text": " no modification happening whatsoever and that's pretty much it I don't want to be"}, {"start": 1913.04, "end": 1917.28, "text": " too nitpicky here I mean the results are fairly astounding even with these"}, {"start": 1917.28, "end": 1924.2, "text": " tweaking I mean in two years time with better hardware bigger models more"}, {"start": 1924.2, "end": 1930.64, "text": " a novel like research ideas this can just get better cool hopefully like this"}, {"start": 1930.64, "end": 1935.64, "text": " video if you did consider subscribing click that notification bell share it"}, {"start": 1935.64, "end": 1955.8400000000001, "text": " with your friends and also join the discord community until next time bye"}]