 Hi, this is Lance from Lightchain. This is the sixth video in our series focused on Lightchain Evaluations. So in the first video, we just kind of laid out why Lightchain Evaluations are important. In the second video, we laid out the core Lightchain primitives. We then talked about how to build data sets. First, developer curated. We built one for the Databricks model, which is discussed in this blog post. So we built a question insert data set. And we showed how to edit that. We showed how to append examples to it. We then also showed how to build data sets from user logs. So if you have an app in production, you're getting feedback questions from a user, you can basically plumb those into a data set. We then just talked through how to use an evaluator, in our particular case a built-in Lancet evaluator, to judge our app which we built here. So we had a very simple app that answers questions about a given context. In this case we plumbed in the blog post and we're using GPT-3-5 to answer questions. We're comparing those GPT-3-5 generations or answers to our data set, which was manually curated. So that's really it. We used a built-in Langsmith evaluator called COTQA, which is very convenient for question answering. So that's kind of where we are. That's what we did. Now let's build on this a little bit. And this kind of gives you a summary of what we just did. Now let's say I want to do something different. I want to find my own custom evaluator. So let's say I want a simple assertion that an answer actually was generated or not. Like, was there some kind of failure in generation of the answer? Is this that to turn something empty straight? Or there can be other reasons why no one fails to produce a generation, right? So in this case, I'm changing it up a little bit. I stole my data set of question and answer pairs derived from the Databricks blog post. But in this case, I want my graders to be, it's like a heuristic, it's a hard-coded decision. It's just like, is there an answer or not, right? It's reference-free, there isn't like a ground truth there. It's just just like is there an answer or not right it's reference free there isn't like a ground truth there it's just like is there an answer or not right so it's more like criteria-based uh grading in this case you can this is where it's a little bit squishy be to think of this more as a unit test it's like a very simple assertion for functionality um as opposed to being an evaluation which is using like an lm as a judge or human as judge this is just like a quick test is there an answer present or? And of course this can be used you know an offline evaluation potentially something like CI right. So it's like a nice test or sanity check that my system is actually working. So how would I do that? So it's actually really simple. This little function is actually all you need. But let me actually explain a bit about what's going on here remember our flow diagram let's actually go back up so there's this run and there's an example so the run is my app basically my app has some output right so the run has this dot outputs which is basically the the dict or the object that comes out of my app and And then likewise, my example, which I pass in Evaluator, also has this dot outputs, which is basically everything that's available to the example. In my particular case, what I'm going to care about is, you know, in typical evaluations, I often care about just the answer. But the point is, when I'm doing evaluation, there's this example object and there's this run object. And these two things are being compared in some way so that's really it so remember i have an example and a run now if i build my custom evaluator i plumb in that example and run just that we just talked about and i can just easily fish out from the run which is my llm app i want to get the answer now let's actually go all the way back up and make sure we understand what we just said. So I'm going to go back to my app. Here's my little app, answer database question open AI. It returns a dict with this answer key. That's where we get that answer thing from. So that key depends on my app. In this case, my app returns, you can see it's a dict with answer in it. That's it. So that's why in my custom evaluator, when I go down here, I just go to my outputs from the run. Again, it runs my app. The outputs contains everything output by the app. In that case, that output is a dict, but it has a key answer. So I'm phishing it out like that. And I do something really easy. If there's nothing there, then it's not answered. Otherwise, it is. As simple as it gets. Remember, I have this data set DBRX. I'm going to run on this. So again, here's my app. Basically, we already talked about this. Here's the data set. We already showed how to build that. This is my new little evaluator is answered. I'm going to create a new prefix for it. Add some new metadata. Kick that off. that should run pretty quick and I can go over to my data set it's already done so okay this is kind of interesting so now you see two evaluations the first one we showed previously using the built-in light chain evaluator now I had this new one you can see it has this new prefix and let's open it up and see so in this case those scores are all one because there is an answer every time. That's it. That's a custom evaluator. It shows up here in Langsmith. This scoring is arbitrary. We can choose different means of scoring. But in this particular case, I just chose one or zero. Shows up here. That's it. Pretty simple. And this is extremely flexible. I mean, this is just a function. It can be anything you want. Again, all you need is the ability to fish out your output. You could do string matching. You could do arbitrary different kinds of valuations. And you can also include, for example, LLMs in this to do arbitrary reasoning over your answers and produce scores. I've used this a whole lot, it's extremely convenient, very flexible and building custom values is extremely simple, particularly once you understand that kind of, that information flow that we showed here, all that's happening is I have an example, I have a run, each of those has this dot outputs thing, which gives you access to, in the case of my run, that output object, in my case it was just a dict with answer, and this dot outputs actually gives you access to everything in this example, that's really it. That's all you need to know. And that's really all I wanted to say about the custom evaluars. It's very powerful and very general. Thanks.
Custom Evaluators | LangSmith Evaluations - Part 6
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video introduces how to define your own custom evaluation logic in LangSmith. 

Documentation:
https://docs.smith.langchain.com/evaluation/faq/custom-evaluators

Code:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb