Ask Me Anything: A Simple Strategy For Prompting Language Models | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this short video I cover the "Ask Me Anything: A Simple Strategy For Prompting Language Models" paper introducing a scalable and effective way to prompt language models.

Using this method they manage to outperform GPT3-175B using the open-source GPT-Neo-6B model!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2210.02441
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:09 Pipeline overview
09:05 Results
09:30 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#prompting #promptengineering #languagemodels
 What's up guys, Alex here. In this video I wanna briefly go through this new paper on prompting called Ask Me Anything, a Simple Strategy for Prompting Language Models. They show amazing results, so let me read it to you straight away. So we evaluate this approach EMA across open source model families such as Neo, Bloom, Opt, and T0, and size is 125 million to 175 billion parameters, demonstrating an average performance lift of 10.2% over the few shot baseline. This simple strategy enables the open source GPT-Neo 6B model to match and exceed the performance of few shot GPT-370B on 15 out of 20 popular benchmarks. So that's huge. And in this video I'm gonna briefly walk you through the idea behind the paper. I wanna also briefly mention that I've covered each of these open source models on my channel. I have the paper covered and I also have a code walkthrough videos of a couple of them. So if you wanna check those out, I'm gonna link them somewhere there in the video card. Okay, so let's see what the paper is about. So they say here, to mitigate the high degree of effort involved in prompting, we instead ask whether collecting multiple effective yet imperfect prompts and aggregating them can lead to a high quality prompting strategy. So as you probably all know, people spend a lot of time finding the exact right prompt that's gonna give them the best result on a particular task. And that's gonna bother some and takes a lot of time. So there is a need to create these methods that are automatic and that can reliably give better performance across a suite of tasks. And that's exactly what this approach does. So our approach recursively uses the LLM to transform task inputs to the effective QA format. We apply these prompts to collect several noisy votes for the inputs true labels. You can see already it's gonna be some form of aggregating the outputs of imperfect prompts. And what they do is not majority voting, they use something a bit more delicate. We're gonna see the details a bit later. But here is the actual high level diagram. So on the left hand side, you can see the input example. So that's the prompt that you would pass, actually tokenize it, you would pass that to an LLM, and then you would sample out the answer to your task. And so let me kind of read it to you. So is the following claim true or false given the context? And the context is the following, John and his friends went to the theater and saw Jurassic Park. And then the claim is John went to the park and the model is supposed to answer that this is a false claim given this context. So one thing you could do is you could basically, well, give it multiple shots, multiple examples of how to handle a similar problem like this one. And that usually requires a lot of, as I said, prompt engineering. So here is the alternative approach that they suggest here. So the idea is to have multiple of these chains. You can see here, they have three chains in this particular image. And each of these chains are gonna have a slight variations in the ways how they could like form these claims and questions, et cetera, et cetera. And well, I can show you that immediately basically. Let me show you what the idea is. So here are some of the variations they do. So for example, so you can vary the in context and demonstrations. Here's an example. So instead of having this claim, Jack camped with Mark, did Jack camp with Mark, whatever, they just kind of put in different claims and questions. And the idea of this one in particular is to learn how to give an acclaim, formulate a question. And you can see, you can just vary that, or you can also just vary how do you form the question. So here, Jack camped with Mark, you have did Jack camp with Mark. And instead here, they vary the question style to wh, so the who, what, where, et cetera questions. As you can see here, they formulate who did Jack camp with or what was not hard. So those are some of the variations that each of these chains are gonna contain. So now let me go back to the main diagram. So once we have those chains, let's see what's the idea. The idea is because of their finding, let me go there. So they find that the prompts that encourage open-ended answers, where did John go to be more effective than prompts that restrict the model output to particular tokens. For example, John went to the park, output true or false. So this is more restrictive as opposed to being more open-ended. And because of this realization, they try and reformulate via this pipeline, the input task into this more of a open-ended Q&A format, that's the idea. So you can see here how they are basically prompting the model to formulate the question. So here is a claim, Jack camped with Mark, and then formulate the question, did Jack camp with Mark? And then another pair, another shot here, and then finally, we take our actual claim from our input example, John went to the park and we prompt the model, hey, give us the question, given the shots that we have shown you before here, okay? So that's the idea, we give you some examples, do the same thing. And then the model outputs, did John go to the park? Okay, and so now, so this is the first stage of this pipeline, the so-called question prompt. And then the second part is, so you can see here, answer the question from context, okay? So context, Joe's birthday was yesterday, then the model is supposed to first output the question, instead of going directly from context and claim into the answer, they instead go through the question as the intermediate step. And that they found out gives better results. Okay, a little bit of alchemy and dark magic here. And so now what happens here is we give the actual context, which is John and his friends went to the theater and saw Jurassic Park, you can see that's the same context as this one is in our actual input example. And then we give the actual output from the previous step from stage one, the question prompt here. And so we plug that in, and then we get the answer. So this is how we get to the actual answer. So instead of just inputting this and giving the answer, we instead first formulate the question, then insert the question here, and only then do we answer, okay? So that's the first part of this method. The second part is you somehow have to combine the outputs of multiple chains. And what they do is not a simple majority voting, which they show underperforms this method. So majority voting would be, for example, if we had three of these chains giving false, true, and false, then the majority voting would simply say false. Well, in this particular example, the output is the same for their method, but as you can see here, they infer the underlying graph structure between the chains. And so basically, because these prompts, these chains are not independent, they try and figure out the dependency and how the output will correlate depending on the changes in the input, okay? Okay, so let's continue here. As I said, so let me read this to you. So we propose the use of weak supervision to reliably aggregate predictions. We find that the errors produced by the predictions of different chains can be highly varying and correlated, right, and so while majority voting may do well on certain sets of prompts, it performs poorly in the above cases, okay? And additionally, worth pointing out is this is not a panacea. There are certain tasks that this method prefers. So they say here, we find the largest gains are on the tasks where the knowledge required to complete the task is found in the provided context, okay? And comparatively less on closed books tasks, for example, factual recall. So the idea is if the answer is already somewhere in the actual context of your input task, then this performs very nicely. Finally, a couple more details I wanna show you. As I said, this is gonna be a brief video. I mentioned this variations here, so they are kind of strategic in those variations. It's not just random variations. They say here to vary the style of open-ended prompt questions, we construct question and answer prompts that produce an answer either yes, no, WH, multiple choice, or close questions. So that's, as I said here, some of these questions are such that, so these are the WH questions, and then some of these questions like did Jack Kemp with Mark or yes or no questions, and they mentioned some other classes of questions that they force these different chains to produce, and that's the idea. Okay, so finally the results are here. So as they said in the abstract, they outperformed GPT-3 few shot on 15 out of 20 tasks, which is super impressive. Again, keep in mind that they are comparing against a model that's against the results published in this paper, Brown et al. from 2020, so the GPT-3 paper. But anyhow, this is super impressive, and yeah, I just wanted to briefly show you this method in case you are not familiar with all of these various advances that are happening in the prompt engineering space. I think this one caught my eye and so thought explaining it to you guys. Awesome, so if you like this video, subscribe to this channel, share the video out, and until next time, bye-bye. And I'll see you next time.
[{"start": 0.0, "end": 2.32, "text": " What's up guys, Alex here."}, {"start": 2.32, "end": 4.62, "text": " In this video I wanna briefly go through this new paper"}, {"start": 4.62, "end": 6.42, "text": " on prompting called Ask Me Anything,"}, {"start": 6.42, "end": 9.540000000000001, "text": " a Simple Strategy for Prompting Language Models."}, {"start": 9.540000000000001, "end": 10.8, "text": " They show amazing results,"}, {"start": 10.8, "end": 13.58, "text": " so let me read it to you straight away."}, {"start": 13.58, "end": 16.28, "text": " So we evaluate this approach EMA"}, {"start": 16.28, "end": 17.94, "text": " across open source model families"}, {"start": 17.94, "end": 20.54, "text": " such as Neo, Bloom, Opt, and T0,"}, {"start": 20.54, "end": 25.44, "text": " and size is 125 million to 175 billion parameters,"}, {"start": 25.44, "end": 28.580000000000002, "text": " demonstrating an average performance lift of 10.2%"}, {"start": 28.58, "end": 30.52, "text": " over the few shot baseline."}, {"start": 30.52, "end": 32.6, "text": " This simple strategy enables the open source"}, {"start": 32.6, "end": 36.56, "text": " GPT-Neo 6B model to match and exceed the performance"}, {"start": 36.56, "end": 41.56, "text": " of few shot GPT-370B on 15 out of 20 popular benchmarks."}, {"start": 43.36, "end": 45.28, "text": " So that's huge."}, {"start": 45.28, "end": 48.519999999999996, "text": " And in this video I'm gonna briefly walk you through"}, {"start": 48.519999999999996, "end": 50.12, "text": " the idea behind the paper."}, {"start": 50.12, "end": 52.0, "text": " I wanna also briefly mention that I've covered"}, {"start": 52.0, "end": 54.7, "text": " each of these open source models on my channel."}, {"start": 54.7, "end": 57.36, "text": " I have the paper covered and I also have a code"}, {"start": 57.36, "end": 59.04, "text": " walkthrough videos of a couple of them."}, {"start": 59.04, "end": 60.4, "text": " So if you wanna check those out,"}, {"start": 60.4, "end": 63.8, "text": " I'm gonna link them somewhere there in the video card."}, {"start": 63.8, "end": 66.92, "text": " Okay, so let's see what the paper is about."}, {"start": 66.92, "end": 69.5, "text": " So they say here, to mitigate the high degree of effort"}, {"start": 69.5, "end": 71.68, "text": " involved in prompting, we instead ask"}, {"start": 71.68, "end": 75.12, "text": " whether collecting multiple effective yet imperfect prompts"}, {"start": 75.12, "end": 78.2, "text": " and aggregating them can lead to a high quality"}, {"start": 78.2, "end": 79.36, "text": " prompting strategy."}, {"start": 79.36, "end": 83.24, "text": " So as you probably all know, people spend a lot of time"}, {"start": 83.24, "end": 85.86, "text": " finding the exact right prompt that's gonna give them"}, {"start": 85.86, "end": 87.8, "text": " the best result on a particular task."}, {"start": 87.8, "end": 89.84, "text": " And that's gonna bother some and takes a lot of time."}, {"start": 89.84, "end": 93.2, "text": " So there is a need to create these methods"}, {"start": 93.2, "end": 96.12, "text": " that are automatic and that can reliably give"}, {"start": 96.12, "end": 99.6, "text": " better performance across a suite of tasks."}, {"start": 99.6, "end": 101.6, "text": " And that's exactly what this approach does."}, {"start": 101.6, "end": 104.12, "text": " So our approach recursively uses the LLM"}, {"start": 104.12, "end": 108.32, "text": " to transform task inputs to the effective QA format."}, {"start": 108.32, "end": 111.76, "text": " We apply these prompts to collect several noisy votes"}, {"start": 111.76, "end": 113.52, "text": " for the inputs true labels."}, {"start": 113.52, "end": 115.44, "text": " You can see already it's gonna be some form"}, {"start": 115.44, "end": 120.44, "text": " of aggregating the outputs of imperfect prompts."}, {"start": 121.36, "end": 123.08, "text": " And what they do is not majority voting,"}, {"start": 123.08, "end": 126.36, "text": " they use something a bit more delicate."}, {"start": 126.36, "end": 129.2, "text": " We're gonna see the details a bit later."}, {"start": 129.2, "end": 133.34, "text": " But here is the actual high level diagram."}, {"start": 133.34, "end": 135.96, "text": " So on the left hand side, you can see the input example."}, {"start": 135.96, "end": 139.92, "text": " So that's the prompt that you would pass,"}, {"start": 139.92, "end": 142.76, "text": " actually tokenize it, you would pass that to an LLM,"}, {"start": 142.76, "end": 145.12, "text": " and then you would sample out the answer"}, {"start": 145.12, "end": 146.44, "text": " to your task."}, {"start": 146.44, "end": 148.88, "text": " And so let me kind of read it to you."}, {"start": 148.88, "end": 152.20000000000002, "text": " So is the following claim true or false given the context?"}, {"start": 152.20000000000002, "end": 153.52, "text": " And the context is the following,"}, {"start": 153.52, "end": 156.20000000000002, "text": " John and his friends went to the theater"}, {"start": 156.20000000000002, "end": 157.96, "text": " and saw Jurassic Park."}, {"start": 157.96, "end": 161.12, "text": " And then the claim is John went to the park"}, {"start": 161.12, "end": 164.20000000000002, "text": " and the model is supposed to answer"}, {"start": 164.20000000000002, "end": 168.84, "text": " that this is a false claim given this context."}, {"start": 168.84, "end": 172.5, "text": " So one thing you could do is you could basically,"}, {"start": 172.5, "end": 176.22, "text": " well, give it multiple shots, multiple examples"}, {"start": 176.22, "end": 179.5, "text": " of how to handle a similar problem like this one."}, {"start": 179.5, "end": 182.14, "text": " And that usually requires a lot of,"}, {"start": 182.14, "end": 184.38, "text": " as I said, prompt engineering."}, {"start": 184.38, "end": 187.72, "text": " So here is the alternative approach that they suggest here."}, {"start": 187.72, "end": 189.74, "text": " So the idea is to have multiple of these chains."}, {"start": 189.74, "end": 191.74, "text": " You can see here, they have three chains"}, {"start": 191.74, "end": 193.26, "text": " in this particular image."}, {"start": 193.26, "end": 197.54, "text": " And each of these chains are gonna have a slight variations"}, {"start": 197.54, "end": 202.1, "text": " in the ways how they could like form these claims"}, {"start": 202.1, "end": 204.46, "text": " and questions, et cetera, et cetera."}, {"start": 204.46, "end": 207.54, "text": " And well, I can show you that immediately basically."}, {"start": 207.54, "end": 210.06, "text": " Let me show you what the idea is."}, {"start": 210.06, "end": 211.62, "text": " So here are some of the variations they do."}, {"start": 211.62, "end": 214.06, "text": " So for example, so you can vary the in context"}, {"start": 214.06, "end": 215.7, "text": " and demonstrations."}, {"start": 215.7, "end": 216.54, "text": " Here's an example."}, {"start": 216.54, "end": 219.06, "text": " So instead of having this claim, Jack camped with Mark,"}, {"start": 219.06, "end": 221.45999999999998, "text": " did Jack camp with Mark, whatever,"}, {"start": 221.45999999999998, "end": 226.45999999999998, "text": " they just kind of put in different claims and questions."}, {"start": 227.18, "end": 229.78, "text": " And the idea of this one in particular is to learn"}, {"start": 229.78, "end": 233.26, "text": " how to give an acclaim, formulate a question."}, {"start": 233.26, "end": 235.18, "text": " And you can see, you can just vary that,"}, {"start": 235.18, "end": 238.22, "text": " or you can also just vary how do you form the question."}, {"start": 238.22, "end": 240.7, "text": " So here, Jack camped with Mark,"}, {"start": 240.7, "end": 242.7, "text": " you have did Jack camp with Mark."}, {"start": 242.7, "end": 246.86, "text": " And instead here, they vary the question style to wh,"}, {"start": 246.86, "end": 250.86, "text": " so the who, what, where, et cetera questions."}, {"start": 250.86, "end": 254.1, "text": " As you can see here, they formulate who did Jack camp with"}, {"start": 254.1, "end": 256.48, "text": " or what was not hard."}, {"start": 256.48, "end": 257.78, "text": " So those are some of the variations"}, {"start": 257.78, "end": 260.78, "text": " that each of these chains are gonna contain."}, {"start": 260.78, "end": 264.38, "text": " So now let me go back to the main diagram."}, {"start": 264.38, "end": 267.21999999999997, "text": " So once we have those chains, let's see what's the idea."}, {"start": 267.21999999999997, "end": 271.82, "text": " The idea is because of their finding, let me go there."}, {"start": 271.82, "end": 274.17999999999995, "text": " So they find that the prompts that encourage"}, {"start": 274.17999999999995, "end": 278.17999999999995, "text": " open-ended answers, where did John go to be more effective"}, {"start": 278.17999999999995, "end": 280.78, "text": " than prompts that restrict the model output"}, {"start": 280.78, "end": 282.09999999999997, "text": " to particular tokens."}, {"start": 282.09999999999997, "end": 285.61999999999995, "text": " For example, John went to the park, output true or false."}, {"start": 285.62, "end": 288.26, "text": " So this is more restrictive as opposed to being more open-ended."}, {"start": 288.26, "end": 292.9, "text": " And because of this realization, they try and reformulate"}, {"start": 292.9, "end": 296.74, "text": " via this pipeline, the input task into this more"}, {"start": 296.74, "end": 299.3, "text": " of a open-ended Q&A format, that's the idea."}, {"start": 299.3, "end": 303.54, "text": " So you can see here how they are basically prompting"}, {"start": 303.54, "end": 305.3, "text": " the model to formulate the question."}, {"start": 305.3, "end": 308.86, "text": " So here is a claim, Jack camped with Mark,"}, {"start": 308.86, "end": 312.18, "text": " and then formulate the question, did Jack camp with Mark?"}, {"start": 312.18, "end": 317.18, "text": " And then another pair, another shot here, and then finally,"}, {"start": 317.46, "end": 320.82, "text": " we take our actual claim from our input example,"}, {"start": 320.82, "end": 323.34000000000003, "text": " John went to the park and we prompt the model,"}, {"start": 323.34000000000003, "end": 325.38, "text": " hey, give us the question, given the shots"}, {"start": 325.38, "end": 329.22, "text": " that we have shown you before here, okay?"}, {"start": 329.22, "end": 331.46000000000004, "text": " So that's the idea, we give you some examples,"}, {"start": 331.46000000000004, "end": 332.62, "text": " do the same thing."}, {"start": 332.62, "end": 335.42, "text": " And then the model outputs, did John go to the park?"}, {"start": 335.42, "end": 338.82, "text": " Okay, and so now, so this is the first stage"}, {"start": 338.82, "end": 342.02, "text": " of this pipeline, the so-called question prompt."}, {"start": 342.02, "end": 344.97999999999996, "text": " And then the second part is, so you can see here,"}, {"start": 344.97999999999996, "end": 347.29999999999995, "text": " answer the question from context, okay?"}, {"start": 347.29999999999995, "end": 350.41999999999996, "text": " So context, Joe's birthday was yesterday,"}, {"start": 350.41999999999996, "end": 352.9, "text": " then the model is supposed to first output the question,"}, {"start": 352.9, "end": 356.78, "text": " instead of going directly from context and claim"}, {"start": 356.78, "end": 359.62, "text": " into the answer, they instead go through the question"}, {"start": 359.62, "end": 361.97999999999996, "text": " as the intermediate step."}, {"start": 361.97999999999996, "end": 365.02, "text": " And that they found out gives better results."}, {"start": 365.02, "end": 367.62, "text": " Okay, a little bit of alchemy and dark magic here."}, {"start": 368.58, "end": 371.38, "text": " And so now what happens here is we give"}, {"start": 371.38, "end": 373.94, "text": " the actual context, which is John and his friends"}, {"start": 373.94, "end": 376.1, "text": " went to the theater and saw Jurassic Park,"}, {"start": 376.1, "end": 378.82, "text": " you can see that's the same context as this one"}, {"start": 378.82, "end": 380.94, "text": " is in our actual input example."}, {"start": 380.94, "end": 383.98, "text": " And then we give the actual output from the previous step"}, {"start": 383.98, "end": 385.9, "text": " from stage one, the question prompt here."}, {"start": 385.9, "end": 389.74, "text": " And so we plug that in, and then we get the answer."}, {"start": 389.74, "end": 391.38, "text": " So this is how we get to the actual answer."}, {"start": 391.38, "end": 394.21999999999997, "text": " So instead of just inputting this and giving the answer,"}, {"start": 394.21999999999997, "end": 395.82, "text": " we instead first formulate the question,"}, {"start": 395.82, "end": 397.34, "text": " then insert the question here,"}, {"start": 397.34, "end": 399.62, "text": " and only then do we answer, okay?"}, {"start": 399.62, "end": 401.58, "text": " So that's the first part of this method."}, {"start": 401.58, "end": 404.82, "text": " The second part is you somehow have to combine"}, {"start": 404.82, "end": 407.5, "text": " the outputs of multiple chains."}, {"start": 407.5, "end": 410.26, "text": " And what they do is not a simple majority voting,"}, {"start": 410.26, "end": 413.58, "text": " which they show underperforms this method."}, {"start": 413.58, "end": 415.82, "text": " So majority voting would be, for example,"}, {"start": 415.82, "end": 418.66, "text": " if we had three of these chains giving false, true,"}, {"start": 418.66, "end": 422.34000000000003, "text": " and false, then the majority voting would simply say false."}, {"start": 422.34000000000003, "end": 423.82, "text": " Well, in this particular example,"}, {"start": 423.82, "end": 425.74, "text": " the output is the same for their method,"}, {"start": 425.74, "end": 428.02, "text": " but as you can see here, they infer"}, {"start": 428.02, "end": 432.06, "text": " the underlying graph structure between the chains."}, {"start": 432.06, "end": 436.34, "text": " And so basically, because these prompts,"}, {"start": 436.34, "end": 437.82, "text": " these chains are not independent,"}, {"start": 437.82, "end": 439.58, "text": " they try and figure out the dependency"}, {"start": 439.58, "end": 442.38, "text": " and how the output will correlate"}, {"start": 442.38, "end": 445.26, "text": " depending on the changes in the input, okay?"}, {"start": 445.26, "end": 447.58, "text": " Okay, so let's continue here."}, {"start": 448.58, "end": 450.9, "text": " As I said, so let me read this to you."}, {"start": 450.9, "end": 453.46, "text": " So we propose the use of weak supervision"}, {"start": 453.46, "end": 455.06, "text": " to reliably aggregate predictions."}, {"start": 455.06, "end": 457.5, "text": " We find that the errors produced by the predictions"}, {"start": 457.5, "end": 460.78, "text": " of different chains can be highly varying and correlated,"}, {"start": 460.78, "end": 463.98, "text": " right, and so while majority voting may do well"}, {"start": 463.98, "end": 465.54, "text": " on certain sets of prompts,"}, {"start": 465.54, "end": 470.46, "text": " it performs poorly in the above cases, okay?"}, {"start": 470.46, "end": 473.42, "text": " And additionally, worth pointing out"}, {"start": 473.42, "end": 475.58, "text": " is this is not a panacea."}, {"start": 475.58, "end": 479.22, "text": " There are certain tasks that this method prefers."}, {"start": 479.22, "end": 482.12, "text": " So they say here, we find the largest gains"}, {"start": 482.12, "end": 484.58, "text": " are on the tasks where the knowledge required"}, {"start": 484.58, "end": 489.21999999999997, "text": " to complete the task is found in the provided context, okay?"}, {"start": 489.21999999999997, "end": 492.53999999999996, "text": " And comparatively less on closed books tasks,"}, {"start": 492.53999999999996, "end": 494.02, "text": " for example, factual recall."}, {"start": 494.02, "end": 496.78, "text": " So the idea is if the answer is already somewhere"}, {"start": 496.78, "end": 500.02, "text": " in the actual context of your input task,"}, {"start": 500.02, "end": 502.94, "text": " then this performs very nicely."}, {"start": 502.94, "end": 504.86, "text": " Finally, a couple more details I wanna show you."}, {"start": 504.86, "end": 506.82, "text": " As I said, this is gonna be a brief video."}, {"start": 506.82, "end": 508.88, "text": " I mentioned this variations here,"}, {"start": 508.88, "end": 511.18, "text": " so they are kind of strategic in those variations."}, {"start": 511.18, "end": 512.98, "text": " It's not just random variations."}, {"start": 512.98, "end": 515.36, "text": " They say here to vary the style"}, {"start": 515.36, "end": 517.38, "text": " of open-ended prompt questions,"}, {"start": 517.38, "end": 519.48, "text": " we construct question and answer prompts"}, {"start": 519.48, "end": 522.6, "text": " that produce an answer either yes, no,"}, {"start": 522.6, "end": 525.44, "text": " WH, multiple choice, or close questions."}, {"start": 525.44, "end": 527.04, "text": " So that's, as I said here,"}, {"start": 527.04, "end": 529.32, "text": " some of these questions are such that,"}, {"start": 529.32, "end": 531.72, "text": " so these are the WH questions,"}, {"start": 531.72, "end": 536.72, "text": " and then some of these questions like did Jack Kemp"}, {"start": 536.76, "end": 538.88, "text": " with Mark or yes or no questions,"}, {"start": 538.88, "end": 541.16, "text": " and they mentioned some other classes of questions"}, {"start": 541.16, "end": 544.68, "text": " that they force these different chains to produce,"}, {"start": 544.68, "end": 546.0799999999999, "text": " and that's the idea."}, {"start": 546.0799999999999, "end": 549.54, "text": " Okay, so finally the results are here."}, {"start": 549.54, "end": 552.9, "text": " So as they said in the abstract,"}, {"start": 552.9, "end": 557.9, "text": " they outperformed GPT-3 few shot on 15 out of 20 tasks,"}, {"start": 557.92, "end": 559.3199999999999, "text": " which is super impressive."}, {"start": 559.3199999999999, "end": 561.16, "text": " Again, keep in mind that they are comparing"}, {"start": 561.16, "end": 564.16, "text": " against a model that's against the results"}, {"start": 564.16, "end": 566.56, "text": " published in this paper, Brown et al. from 2020,"}, {"start": 566.56, "end": 568.12, "text": " so the GPT-3 paper."}, {"start": 568.12, "end": 572.16, "text": " But anyhow, this is super impressive,"}, {"start": 572.16, "end": 576.0, "text": " and yeah, I just wanted to briefly show you this method"}, {"start": 576.0, "end": 577.92, "text": " in case you are not familiar"}, {"start": 577.92, "end": 580.44, "text": " with all of these various advances"}, {"start": 580.44, "end": 582.5600000000001, "text": " that are happening in the prompt engineering space."}, {"start": 582.5600000000001, "end": 584.24, "text": " I think this one caught my eye"}, {"start": 584.24, "end": 585.88, "text": " and so thought explaining it to you guys."}, {"start": 585.88, "end": 587.76, "text": " Awesome, so if you like this video,"}, {"start": 587.76, "end": 589.8, "text": " subscribe to this channel, share the video out,"}, {"start": 589.8, "end": 592.36, "text": " and until next time, bye-bye."}, {"start": 592.36, "end": 597.36, "text": " And I'll see you next time."}]