 Happy New Year, everybody. Thank you for joining. This is a Graph Geeks Explainer talk. My name is Amy Hotler, and today we're going to be talking about how you build knowledge graphs that can process millions of new entities and relationships every day. We're very fortunate to have Rob Kalk, founder of Emergent Methods with us. And you'll also be learning about how their team enabled what they're calling on the fly sub graphs, which is very exciting and helps you kind of enable more targeted domain specific expiration of news. I'd like to turn it over to Rob to take it away. And I think Rob, you have a poll for me, don't you? Yeah, let's start the poll. Hey guys and gals. My name is Rob. I'm the CEO of Emergent Methods, like she said. And we have a mission to distribute real-time information using the latest tools, especially focused on some applied research. Our background is in academia and research. So we bring our approach to hypothesis conception and testing and identifying, you know, really important conclusions. We bring that into all of our work, our software and our products. So today we're going to chat about exploring the news at scale, which is kind of our claim to fame, the largest news knowledge graph on the planet. And we haven't found anyone that has debunked it yet. I'm perfectly happy for someone to come challenge us. But it's true that we're upserting a million entities a day into this thing. And that requires some very creative structuring of the data, as you might imagine, and you'll find out shortly. But let me just get a sense of first, how everybody describes themselves so I can tailor the presentation, okay? Looks like we're majority developers, and zero citizen scientists, which I'm sad about, because I think everybody is a citizen scientist. So I'll just put everybody in the secondary category of also citizen scientist. But that's great. So let's get started. First, I just want to say the work here is a lot of a team effort. So there's Ellen Tornquist and Wagner Costasantis out of Brazil. These are really smart people who work really hard in the trenches, and this would not be possible without them. So let's see if we can move forward here. Ask News. I just want to give you a brief overview of what Ask News is so that you can give it so that you have context to why we're building this thing or how we have that type of information at our fingertips to build the largest news knowledge graph. We're tracking thousands and thousands of news sources, open web across hundreds of countries and many languages. And we're applying all of the latest tech, injecting our own research, especially on entity recognition, and obviously obviously the graph extraction which we'll talk about soon i'll say a lot of what you're going to see here is actually not a graph database but instead it's a vector database there is some sprinkling of graph dbs around and we'll talk a little bit about that if the questions head that direction. But yeah, Quadrant is a big piece of this. So what's our general goal? We want to provide high quality data to real time information to pretty much anyone, citizen scientists, analysts, and LLMs. How do you communicate that information? First, how do we get it? And then how do we communicate it? A lot of this is content enrichment. I'm not going to dwell on this because you know i want to move to the more interesting stuff but generally speaking there's just a lot of enrichment extracting entities extracting sentiment classification a lot of the traditional stuff that we used to do with older tools are now having higher quality and a lot more customizable to a domain specific stuff, you know, talking about extracting statements and evidence and attribution. You couldn't really do this before you had a large language model that could read the text and say, hey, here's the statement. Where's the evidence? Where's it being attributed to? That sort of stuff is some extraction and enrichment. enrichment. You can head over to asknews.app and read our copy. We have an editor in-house, so this is human in the loop stuff. We take it very seriously, and we believe we're the least biased source on the planet because it's an algorithmic enforcement of diversity, and we take pride in that. We love our our sentiment analysis we're doing all sorts of stuff evaluating provocativity extracting the entities etc etc we have a claim to fame which is gliner we didn't build the architecture but we find to engineer the data set to fine-tune the data and that's kind of the start of building out a graph is how do you actually get the metadata to create a graph, right? You need to extract that data properly. And can you extract Eastern names and African names and South American names and Western names all at the same time? That's a big piece of reducing bias in your index, right? And by the way, if uh if you want if you have any questions i'd love to you know clarify things and make sure everybody's on this page um okay so let's move forward here entity relationships right that's what we're here for uh let me stop droning on about Ask News. Every graph is different. You always have some different application. In our particular example, we're trying to explain relationships of text and communicate that to either an analyst or an LLM. That's a big piece of the puzzle that we're working on. And so that means that the relationships are not necessarily predefined. And that's where you enter an ontology-free approach to a graph. You can see the relationships are not predefined. They are nearly unlimited. So what does that mean? Here's another example of extracting a graph. I think everybody's here at least comfortable with graphs, but I just want to make sure everyone's on the same page that really we're just trying to find out how are people related to locations, locations related to organizations, even things like an item like a shake. Sally bought the shake and Roger bought the shake. When you extract this information, all of a sudden you've represented that in a very different way. And you can communicate that and store that information into what we're calling the largest news knowledge graph on the planet. But you can communicate that to an LLM or to an analyst and it almost opens up other insights. So let's see if we can move forward here so how did we start right that's kind of one of the questions that we're answering in this presentation we needed to start with well we're going to process what uh you know a million news article texts per day that's a lot of text and putting that all through openai gpt4o So that is going to's a lot of text. And putting that all through OpenAI GPT 4.0, that is going to cost a lot of money. It might work pretty well, but we're a tiny startup. We're not able to devote those kinds of resources. So instead, we said, let's see if we can fine-tune something. Selecting model we have phi3 it's really powerful base tool base tiny model it sits on your customer consumer grade gpu which is really impressive so that's lightweight easy to fine-tune open weights a pretty decent size context window 4k all the way up to 128k tokens all that means is you can throw a lot of text in text in and get a lot of reasoning out so the problem here is you know we want to take five threes it has this really great base ability of reasoning. It can extract relationships. It can take a text and it can see that Roger and Sally both were at McDonald's together and they did this and McDonald's was open on Thanksgiving. But we want to tune it so that the output is structured and capable of representing our expressing our vision of how this news text should be graphed. Right? There's a lot of values there that we want to express, the least of which are kind of, you know, journalistic integrity. We don't want to say the wrong relationship. We don't want to say that Sally killed Roger. That would probably be incorrect information. We don't want that in the graph. Maybe that's an extreme example, but we do want to make sure that we're fine tuning the model for our values. So we need to engineer the data set. And that comes down to engineering diversity, to be quite frank, right? we want to make sure that it is seeing all sorts of examples of what we would want it to be able to extract in the future so you don't want to you know hope you want to hope that uh that it has it can extract a graph on names from south america if it's never seen them, it's not going to do as well. But if you've already injected that, engineered that data set, you're now making sure that, hey, at least it's aware that these types of names exist. You're building out what's called a parameter space when you do this. And so really the goal of engineering your data set is to cover that parameter space, cover the linguistic diversity that you're going to encounter, cover the geographical topic diversity. Do you want to be able to recognize only sports, sports, politics, and finance? That's a key piece of the puzzle. And so, you know, I don't want to dwell too much. I don't want to but I want you guys to be aware of how the data was created and how it was engineered. We have a very high quality synthetic data set here in all of our stories, which are human in the loop edited, right? That's our we have our editor in chief in house who's made sure that these stories are being written with our journalistic integrity. So we want to fine tune on text that we find to have journalistic integrity so that it's building graphs that have standards. We extract. You go ahead. Rob, there are a couple of questions that have come up. Oh, no, no, this is fantastic. There's a first question is from Mitesh. I would like to appreciate your time, but would like to understand how you handle contradictory information in the news. And, you know, or if you're going to cover that later, we can talk about that later as well. So that's the first question. Yeah, that's a great question. It's a little bit of a different topic, but just as a short response, we enforce diversity from different languages and countries when we report on a large topic. So if it's Israel, Gaza, we're not going to just only take sources from Israel. We're going to say, OK, source from Israel, source from Gaza, source from Germany, Sweden, America, and that allows us to identify the contradiction, identify the alignment, and report that. So we're not actually saying what is fact and what is fiction. We're saying this is what is contradicting, based on understanding how different languages and continents are reporting on it. Yeah, there's another question from Prashanth, which is maybe a little more related to what's on the screen right now which is um how are the verbs in the relationships decided yeah so um that is that goes back to a couple points this is the reason this is the llm's ability to detect what is going on so we're not pre-defining relationships that's a key piece of this puzzle we are extracting information in a graph graphical way based on what the LLM is capable of reasoning so the LLM is deciding these okay what I was just getting to here is building the synthetic data which is your training set. We agreed that Phi-3 is kind of small and maybe a little or a lot dumber than something like a Claude 3.5 or a GPT-4. Those are very smart, large models that have a lot more intelligence behind them. So they're able to detect better relationships. You could say better verbs in some way, but generally speaking, just better relationships extracted from the text. So we actually leverage that intelligence by using GPT-4.0 to build that synthetic data set. Here, let me see. We have another question. How stable are the cross comparison of sources and how do you decide what is passably coherent? I like these questions. I think they're off topic of the graph. Should I tangent off? Do we have time? I'm happy to answer it. I just feel it's a little different. I mean, Jason, let me ask you, Jason, does it relate to the graph? Can you maybe clarify why it's related to the graph construction? So I would say, Rob, we can proceed on the graph side of things for a little while, and then we can always circle back uh about um you know the you know the the how you're extracting like sentiments and things like that which i feel like a lot of these are personal related so yeah yeah i mean if it's quickly before or maybe we yeah we can move forward to this one um towards the end or whatever section it fits into yeah no it's a good question i just but it is slightly tangent um okay so we'll extract that information using a smart lm like gpt4l so this is building our training data set our eval data set and our validation data set which is an important aspect in machine learning when you're trying to properly train one of these models. So the beauty of this is, hey, you can, as long as you've established and painted your parameter space properly across language and topic and country of origin, now all of a sudden, you've done this with a very smart model, you've established your training set, you can now train PHY3 and it's going to emulate how GPT-4.0 was making its decisions when it was extracting those relationships and extracting those entities. So that's really powerful, because what you've done, for all intents and purposes, is a knowledge transfer from a very smart LLM to a very small LLM. For the actual fine tuning itself, we take that data set and now we've we use pretty easy tools right now the the abstraction level is extremely high and very easy you know what's underneath this these like 10 lines of code here is probably million around the on the order of millions of lines of codes that are of code that's all being leveraged. So I really encourage people to explore. And if you think you can't find, you probably can. It's easier than most of the analytics you've done, for sure. The tools are really easy to use. So the actual fine tuning parts, the easy part, it's the engineering, the data set. That's the hard part, right? You can play with a bunch of these are these parameters. So these are just parameters that you can choose to. And so there's a bit of an arc here, a lot of trial and error, and really with a human eye looking at the output. And that's what we're going to look, talk about right now is, you know, it's one thing to have a metric, which says, hey, this worked well, this was trained correctly. And it's another to just look at the output and say, hey, does this smell right? Does this smell how we're expecting it to smell? Especially when you're working in an ambiguous world where a graph for a text, you could probably create a near infinite number of graphs. Yeah, we'll share the slides for sure, Max. Thanks for asking. You could probably create a near infinite number of graphs. Yeah, we'll share the slides for sure, Max. Thanks for asking. You could probably create a near infinite number of correct graphs for one set of text, especially if it's two paragraphs or three paragraphs, because there are a lot of right ways to explain something. With that said, we can still identify some metrics that can help us say, yes, this is closer to right, and this is a lot wrong. This is called essentially a loss function, and that's what you use when you're trying to tune the weights of that model. So what you're doing is you're applying a small set of your training data set, you're running it through your model, and then you're how wrong was this based on these metrics and we created these metrics out of necessity and this is a more or less a json similarity how close is it to a correct um to the json itself the fields and the values are they close to what the right answer is and the consistency so this is. Is it, you know, are these edges even right? Does this edge make sense? Is this connecting to actual nodes? Are these random? Because LLMs, as you well know, are pretty random sometimes. So this is us trying to train it in a place. You can go check out the model. It's open source and it's also Apache 2.0. Or no, sorry, not this one. Gliner's Apache 2.0. But it is open source. You're welcome to use it for academic purposes. If you want to use it commercially, you can come talk to us and we'll try to help you out. Let me see if I can get to the actual comparisons here. I was talking about how you could make an unlimited number of graphs per text this is where hey we would train it then we look at the output compare it is that to the to the right to the right answer how does that look in this case this is great this looks really good it's really close you've got a little some differences right like in this example silly barily Barakov fled from, but in this case, it was Vasily Barakov. The crime location was in, I can't pronounce these names, I'm sorry, but they're both suburbs of Moscow. So you can see there's some differences, but generally speaking, it's a good representation of the information. But what about when it's just ambiguous, right? The GPT-40 output here versus our GPT-53 mini instruct graph output. Honestly, I prefer ours because it's a bit more structured. It's more higher fidelity. This one is kind of lazy, in my opinion. This would be if I asked an intern, hey, make and they're like yeah everything's related to this right it doesn't really that does that's not compelling me that doesn't make me feel like i'm building a great graph um and then for example here right like this is a bad loss because we had a 57 json consistency um with the output of cloud 3.5 on it but we we had 100% JSON consistency with ours. That was fine-tuned. And that's an indication of, okay, this thing is creating JSON, that's consumable. That JSON can then be structured and stored and used in the future. Yeah, I really like this question too. So when you're engineering a data set with your train, your eval and your validation, you want to try to avoid as much data leakage as possible. Sometimes it's really difficult to avoid data leakage. And what that means is in your eval, you have some information from your training. And this can be as simple as, hey, Trump, hey, Trump won the election. Okay, it's in the training data. And then in your eval, there's some something, some eval about Trump winning the election. It's another text. And you've seen it, your model has now been trained to know that Trump won the election. So it's a little bit of a cheat. So in order to chop that, we used temporal awareness, which goes back to the question from Steve, I appreciate, which is tracking an event through time. And so we're able to cut it off and say, okay, the training data set is only going to be between this date and this date. And then the eval set is between this date and this date. That is a segregation and avoiding leakage. So yeah, so the temporal segregation is really important, and I think it's one reason that we have great extrapolative abilities. And it even goes beyond that. We even just segregate a whole topic. So we'll say, hey, this was a topic that was written about 200 times during the month of July, that is in the training data set, and then we're not going to go talk about that topic ever again. So maybe Ukraine, Russia is mostly only in training. But then, you know, we want to be able to extrapolate out in in eval and make sure that the eval is capable of extrapolating on the future stuff that's happening in Ukraine, not necessarily what occurred during July. Okay, so these are some performance comparisons. We'll kind of move through because the demo is the fun part, but generally speaking, we do better than Claude 3.5 Sonnet and better than the base model, which would make sense, right? We've literally trained the thing to be better than the base model. But Claude 3.5, to run that, you're looking at somewhere on the order of $3,000 a day, $3,700 a day. But if you want to host our model, you're going to pay less than $100. So a massive cost savings. You probably also will have rate limiting problems if you're using OpenAI or Cloud 3.5 to run this. So there's a lot of... And then by the way, you're also sending all of your data to OpenAI. So that's just on their servers. So there's a lot of benefits then by the way you're also sending all of your data to open ai so that's just on their servers so there's a lot of benefits of fine-tuning here let's see let's look at it in action here's an example of israel gaza um here you know this would be kind of the extraction i don't we don't have the let me see if i can pull out the relationships yeah here we go this is. This is what it would look like, selecting only the Islamist fighters here, and then going through and saying, okay, these are the relationships we've extracted for this subpart of the general subgraph. And so this is really good for communication to an analyst. When the analyst says, hey, I want to bring myself up to date on top on this topic i can very rapidly click okay what's going on with islamic islamist fighters and then rapidly consume this information to find out some ex something else like who is hassan abdel ghani you know apparently he leads the islamist fighters but i want to know more, and I can click, right? That's a communication of information, a rapid communication of information, a rapid exploration of a subgraph. So, in some ways, the design of this comes from the consumption of it. At the same time, this is a great way to pass context to an LLM. Let me see if we missed an important question here. As the graph grows this large, how are you preparing for future bottlenecks in query performance to retrieve against specific node or relationships? So structuring, the underlying structuring of this information is one of the most important aspects of it, right? So what you have is a lot of information that is not stored in a GraphDB, because if you're constantly upstoring millions and millions of nodes and entities to a GraphDB, it's not going to work. And so instead, we are indexing this information in a much more traditional way, to be honest, on S3 in some ways, right? So we're indexing it to S3, we're using Quadrant very heavily for a lot of aspects of this. There's a hierarchy of, there's a structure of how we're passing information and indexing information. Now, that doesn't mean we won't use a GraphDB to still do a query because luckily you can still build a quick graph, which is kind of what you see here, right? This is a quickly built graph that was built on the fly. You can bring that graph and put it into something like a memgraph to start doing more local, more traditional graph traversals. And I talked about this with Demetrios over at on the mlops podcast so i would refer you there to get even more details but generally speaking uh that's one of the approaches but yeah there's just a ton of information and in the end you have to store it on in some uh a more creative way the last thing i'll say before moving to the demo is is the kind of variety of people that are using it and what they're using it for. One of our favorites is Utexas because they're using this for detection and classification of misinformation, which we find really, really powerful. These researchers reported a 24% increase in accuracy over old methods of misinformation detection by integrating Ask News and integrating this very high fidelity context. um and then the other one i'll just point out which i'm sorry we like to brag about it uh metaculous a lot of the the bot builders the winners ended up using uh ask news to help provide context to make more accurate forecasts, and they won money. And so what I really want, the picture I really want to paint beyond on the fly graphs and this awesome large knowledge graph is, it's about context engineering. This is context engineering at its core. And if you take the right steps at every point and you do a really strong engineering of your context, you can leverage the intelligence of the LLM at the end of the day in a very powerful way. But you need to paint the context correctly to get the accurate output. And so a lot of what we're doing is just getting you, you know, nine steps of the 10 of the way there so that you can then take that high fidelity context, include it with some of your other pieces of information, and then make some business insight or, you know, combat human trafficking. Here's the team, by the way. And so these guys are really, these guys and Gal are really, really important. So let me jump over here. I made a little notebook. I'm happy to share it with you guys. And I just wanted to kind of go through how this sort of thing can be used. I think one of the, the query that I would love that I want to investigate with you through this demo is understanding the negative media attention, right, of a company or a person, and then trying to figure out why it's happening. And then leverage, again, leveraging the high intelligence of these really powerful LLMs to make recommendations to ameliorate that, right? Because that's kind of the whole gambit. You've got your context for what's currently going on in the media. Is it negative? Great. Do your search, build that graph, filter on the negative nodes, filter on the technology-based nodes, but then grab it and then build out that high-fidelity context, hand it to an LLM, and then get some actual insights out, because that's what people are actually after. People don't really care about the graph. They want to actually get information. And so that's kind of what we're going to walk through here. So, okay, we've got dependencies here. It's just, we're just installing Ask News and Langchain and RapidJSON. We have some helper functions. You can come check those out. These are just to help color the graphs below and stuff like that. They're nothing too important. This would be our client for Ask News. And so you just import it, add your secret and define your scope. And now you can get all the context, build graphs, do whatever you want. Here, this is how this is the call we've used to build the graph. In this case, we're filtering on only negative news in the category of technology. And we're going to look back in this case, 10 days. But we could change this. We could filter on a lot of different ways. We can slice and dice, I think 20 different ways. So I'm showing you three here, but there's a lot more. We could add a lot of our clients do something like, you could do something like this, or even if you want, there's a lot of value in checking only stuff that's sensational. But we won't do that for this particular one. Okay, so I pre-ran this so we didn't have to wait. This might take between 30 seconds and 5 minutes, depending on the number of entities and the number of um uh articles and so you know as you as you can tell a lot of this is disambiguation related and you guys probably know paco from zenzeng they're doing a lot of powerful disambiguation as well we're focused on disambiguating news related entities and understanding how to do that. And that's a few months of research from the fall where we really dove in and said, how do we do this the right way? So this call does a lot for you. This is an on the fly construction, going to S3, finding the right stuff, pulling it out, putting it actually into a GraphDB, doing some traversals, and then getting you the information. In this case, I want to visualize it with Cosmograph to show you guys what's there. So let me see if I can open that up really quick. Cosmograph, really powerful. This is a really cool software which allows you to also visualize the fly and convey information. So just as a reminder, this is all of the, this is the top 500 negative technology news articles and their relations to one another from the last 10 days. You can see Apple here is apparently a larger node, but you can see how the chains, the chain of relationships move through. Cosmograph's really great. Let me move back here. So that's the data that we just we brought in. Here we're going to, we want to find the most important one in the graph, and so you can just go grab it. So this is pretty traditional. You can say, hey, we're doing it in Python here. You could do it in Memgraph, right? We can upload this to Memgraph, and also you do node traversals that way. Here, it's a notebook. I wanted to make it light running Memgraph. You need a Docker container running, but generally speaking, it's more or or less the same thing grab the the node with the most uh or basically count the entities for from uh the relationships for all of the entities uh here we're just going to grab the most important one and it looks like we grabbed apple now we could look through a couple of them to see what what else is is out there besides apple let's if there's, so there's Microsoft. So I'm just kind of going down the list to see if it looks like Apple has the most, Microsoft has the second most. And we're just looking at tech company here. So this will stick with Apple here, but we can also go look at people. We can look at kind of whoever we want in this negative sub graph that we've created right you could even go find your company if you have a particular company but in this case we're just going to take the most negative press from the last 10 days um now we're going to do a little bit of context engineering together okay the idea is let's grab the relationships up to x hops away from the most important entity, because in order for us to understand why this negative news is occurring, I want to get all of the news that is within one hop or two hops or three, whatever you want. I think three is probably pretty far. It's, but at least you're getting a big, you know, contextual picture, even of information that's very tangentially related, it may still play an important role in understanding why is everything so negative around this company. Direct relationships are obvious. Secondary relationships are less obvious and tertiary, maybe useful, maybe completely useless, right? So we grabbed those relationships here, and that's all we've done. We've just filtered them out. Looks like we got 102 entities here. We visualize it with NetworkX, and it looks a little messy, but yeah, App is at the center of this thing. Cosmograph is a much better way to visualize it, but this is in your notebook here. But this would be kind of your sub-subgraph just around Apple that we've extracted. Here we're going to say, okay, let's grab them and get some LLM input. So when we're doing that context engineering, again, we want to convey as much information, information to the llm as possible when we to ask it to generate that business insight so we have the articles we just iterate through the articles and then create a string so it's just literally a string that looks like this right so you can see you got the summary you've got the published date you've got the the title the english title and the original title we could add more metadata here if you want we could add the entities we've got the summary, you've got the published date, you've got the title, the English title and the original title. We could add more metadata here if you want. We could add the entities, we could add the sentiment. You can really go crazy with your context engineering. But generally speaking, this is the string that we're passing, that we will pass to the LLM. But we're not done yet. We have one more piece of context engineering that we'd like to do to pass to this to this LLM and that's to get the graph out. And so what we do here is we just grab the graph. First this is for network x just to make the graph but here we're again we're making a string for the graph for the LLM which is the graph. And so that's going to look something like this. And this is, as you can expect, a very, very primitive, simplistic representation of information. But again, this is highlighting the most important relationships in that subgraph around Apple, which are allowing the LM to continue solidifying its context around the negative media situation. So that's what this string looks like, as you can see. Again, we plotted the network x graph, and so now we're going to actually pass it to an LLM to get the answer. Let's get the, how do we ameliorate the situation for Apple? That's our, that's our goal. I chose to use LangChain because it's quite readable in this case. So here we're just pulling Langchain. We could use Claude or we can use OpenAI. It's kind of up to you. It makes it easy to swap between the two. So we pull that chat anthropic client. Here's our prompt. So, you know, again, this is more context engineering. This is kind of a basic one that you're welcome to start from, pull from. But generally speaking, it's pretty straightforward. We're just saying, hey, here are the steps that we want you to follow based on the information that we're going to give you. So we're going to give you a set of news articles, a graph, and we want you to analyze the news. We want you to analyze the news. We want you to analyze the graph, identify the key issues. This is kind of chain of thought prompting, if you will, which is improving the probability of the next produced token by forcing the LLM to speak the way of the things you need it to speak of in order to draw conclusions of what follows. So that's a mouthful, but essentially what you're doing is you're just saying, hey, in order for you to actually understand how to get me recommendations, you probably first need to have talked about the key issues. And that will make the probability of your next token when you start talking about recommendations much higher because you've you've kind of pigeonholed the llm into you forced it to think and talk about the right things if that makes sense let me know if you have questions i'm happy to to chat about it it's kind of an art it's kind of fun it's context engineering and we just say hey uh write your report and then we just give it a little outline of the report. You could change this, add whatever template you want. These things are really, really flexible. So that's the system prompt. Here's the human prompt that we give it. And this is where we inject the article string and the graph string that we created earlier. So you can see here, this is where the entity will go, and this is where the use article string goes, and this is where the graph string goes. So we've engineered the context, we've told it how we want it to think, how we want it to report, and we've said this is the information that you have at your disposal to actually generate that information. Then you can call it. This is a Langchain syntax. This is the report prompt that we just created right here. And then we chain it to the LLM. And in this case, our LLM is that chat anthropic. And then we invoke it. And this is where the entity gets put into the entity spot. This is actually a glorified f-string of's going on. And so now we've engineered the context and all we need to do is call the LLM to generate that output. Google Colab makes it a little hard to actually see this. So what I'm going to do is switch to a Google Doc. Let's see if this wants to... Here we go. And then get that output, right? And so what it's done is it's read all of that news and taken that graph and given you exactly exactly what we asked for an overview of what's going on uh the the issues so it looks like currently apple's facing some privacy violations there was a secret logging of a calculator app without user user consent siri recording private conversations i read about that one a couple days ago yeah it looks like they are they definitely have negative surrounding them right now. Compliance issues. Okay, this is great. Now, what are the possible outcomes? We asked for it to kind of... Oops. So my iPhone... That's hilarious. So my iPhone just heard me talk about Siri eavesdropping and just started talking, just started to answer a question. So it's been eavesdropping on the entire presentation, just so that we're all on the same page. That's the most ironic thing that's happened to me in a while. Okay, so I won't say her name again, she who will not be named. We talk about the possible outcomes and and then make some recommendations. And this is where we're trying to leverage the LLM intelligence. I'm not a media expert, but I know that the LLM has been trained on a lot of media textbooks and a lot of content talking about how media relations work and how you can help avoid bad press and improve and proactively improve your future press sentiment. And so we just give this to the CEO and then we run, we head on our way. So that's kind of a general approach to context engineering with the graph, with the Ask News SDK. Let me see if I can answer some questions now. There was one question earlier about whether end users can edit the local subgraph. Yeah, you've got the data at your disposal. You can do it kind of with it what you wish. So if we If we go doc you can absolutely edit it and that can come in the form of um using it with memgraph to edit it uploading it to quadrant to edit it or editing it with python you can do whatever you want with it you know we stop right at the point of giving you a graph with a bunch of metadata in a structure that we believe is useful. But we have clients absolutely take that and run with it and do kind of any, they do the last mile, more or less. Okay. I did have one other question about how developers are using Ask News. So it's pretty clear how a researcher would use it. But how are some of the ways that developers building solutions are using it? A couple. So at any point where your application requires some real world context. So, for example, a finance app where they've got ticker data, they've got a lot of their own data sources that they're using to help their users track portfolio analytics. Maybe they want their user to ask a question about their own portfolio. Well, it's great to have ticker data. It's great to have some of your own internal information, but what about the news surrounding that company that extends beyond finance, right? Like we just saw here, probably a lot of this news related to Apple is political. And so understanding that can really make a difference in your portfolio analytics discussion. And so it's helping add that corner of context into their app. So that's one example. So go ahead and ask your question, Steve, if you're having trouble with that, I would say drop it in the chat if for some reason Mike isn't open yet. Can you hear me now? Got it. Okay, cool. Thanks so much. Great session today great really great session today rob yeah i guess i did want to ask have you as your team encountered like especially um like any challenges with trying to detect synthetically generated content on websites versus you know any uh distinguishing felt like sort of more authentic because i Because I know we've definitely seen that in terms of, you know, sort of the mis- and disinformation angles, just lots of regenerators, synthetically generated. So I don't know if that's been something that y'all have had to struggle with. Yeah, that's an awesome question. There's a lot to unpack with that. Our website is in some sense synthetically generated information. And so, yeah, it's more about how the quality of that synthetically generated content. And we have a quality control pipeline, which aims to assess the quality of what's coming in. It's not perfect. It's not only for detecting synthetic content. Actually, we don't even try to detect if it's synthetic. But a lot of what we do is when we're actually deciding what to go out and look for, we look into the website and we say hey what's is this thing a legitimate website producing open web content that is has some value if not then we just don't go and so there's plenty of times where you know i think i i found what we'll we'll spot them ourselves sometimes we'll say oh that yeah that website like one of them leak it it just regurgitated its prompt and it was just clear that they didn't have control over whatever was going on over there so there's that but there's plenty of low quality human-made information as well and so um that's one of that's a big a big piece of this is is diversifying the context and making sure that you're not just pulling from one area. So if you ask about Apple and its privacy violations, you're getting 30 points of view on that. You're not just getting one. I think that's a big piece of it, and that allows you to really at least sequester the lower quality information. Yeah, sounds great. Thanks so much. Appreciate it. Thank you so much. Appreciate it. So Rob, there might be an adjacent question here from Kevin, a curious attendee. Following the idea of garbage in, garbage out, if I understand you correctly, the graph entities and relationships would depend upon how reporters define those. So different outlets countries cultures reporting on the same story you're getting different results and possibly a different graph so depending i assume um kevin the question is depending on how you filter you might end up with different sub graphs as well and how you deal with that yeah um it's a great question and so you can purposefully bias your search right you could actually say i only want sources that originate from south africa and then if you only trust south african sources and so then you'd get only a graph that's built from that at the the same time, you can enforce the diversity. We have a setting where you can say diversify sources, where we go cast a net and try to guarantee that you have the distribution of countries and languages that's present in this general cluster that you're looking at. But there's no way to guarantee that anything is right. Okay, so I think if the question is, how do you make sure that the relationship is correct, you don't, you can't, but you can, I think, again, it's about ensuring that you have a large enough coverage of this reporting that you're comfortable with, okay, generally speaking, the alignment is here, the contradiction is here. And so as long as you trust the democratic approach to the reporting, then you have a better picture of the news. But you cannot eliminate errors and you cannot eliminate the bias even. Even us as a company are injecting our own bias into how these relationships are formulated in some ways indirectly based on how we engineered the data set. I think if we look at... Let me see if I can bring this back. If we look at the data set engineering itself, here's an example of some diversity, which would certainly impact how relationships are extracted. We don't have a great representation for Zambia here. We ensure there's some, but there's just not enough information. So this is kind of our approach of balancing. Hey, we need a lot of training data. We don't want to over overemphasize certain countries. We want to guarantee some representation of some countries. So that's another approach to consider. So we, and we have two, thank you for addressing that. And then related to engineering, actually, we have two data engineering questions. First, I will actually take the second one first, because I think it'll be quicker, is whether you're using glider for the triple extraction. So that's the first question. Sure. And I'll just wrap. I think maybe that other question got edited to the graph does not contain the truth. Okay, so that's an important aspect of what's going on. The graph is just a relation, a picture of the articles that you you selected and what relationships were present in those. So it's not actually presenting any truth or fact or fiction. It's just a graph. With the asknews.app front end, we get closer to a truth. We still don't say anything is fact or fiction, but that's where the graph is a component, but it's not the end-all context engineering. You still need to present the information itself, the actual text, in addition to the graph, to say, hey, what's going on? What are the key alignments? And what's a contradiction? Just to make sure I finished Kevin's question. Yes, we're using Gleiner. Well, okay. We absolutely use Gleiner for entity extraction, not for triple extraction. So the triple extraction is the Phi-3. So that is what Phi-3 is doing more or less. And I think this is essentially what it looks like when PHI-3 extracts information. Thank you. And then probably another, yeah, if you want to look at this other question from Sri, question from Engineering Aspect. Once you receive the information, the data from a few different sources, can you please walk through the pre-processing you do to eventually store in a quadrant and elsewhere before your app takes over yeah uh there's a full pipeline and i probably won't even remember all of the steps there's a lot of them um but you know this this goes to their sporadization there's extraction i think we had a lot of this here. This is a good view of it. Extracting statement evidence and attribution. This is building essentially the synthetic representation of that article. So yeah, extracting the statements, extracting a small summary, extracting the entities, extracting the graph sentiment, reporting voice, this is all processing that information before we even start looking, before we up-sert to Quadrant, right? So, this all happens, and then we up-sert to Quadrant. And actually, a lot of what's stored in Quadrant are just UUIDs in some way, right? Because Quadrant is actually quite scalable and very very powerful um but you don't necessarily want all of your your text inside of quadrant it's sometimes better to just keep uu ids and then reference those uu ids to something like an s3 because s3 is going to be cheaper and more accessible and so you know that know, at least that's how we've approached a lot of the underlying data handling. But does that answer your pre-processing question? Yeah, thanks. When you said about using S3 as a sort of cheaper option and using uuids can you can you please elaborate on that um yeah i mean essentially so a common technique in engineering with especially with databases is to index uuids and then the uuid references some information somewhere else on the web. That could be an image, right? That could be a video. That could be... Like I guarantee you when you're searching in YouTube for videos, you're basically interacting with UUIDs and then when you actually want one, then it streams from somewhere else. And this allows you to be a lot more efficient with meta... You're basically compacting your metadata, you're attaching metadata to a UUID. And that UUID can be used to go fetch a larger amount of data from somewhere else. So that's more or less the approach. And it works very well. Although, to be honest, Quadrant can still store quite a lot of information. So it's kind of a balance. It depends on your app, on where you want to put the information. In some sense, storing your information somewhere else can be cheaper. But then you have another failure spot. So if that connection fails fails you have to make a secondary call that adds latency so you know but in some ways it's more resilient because you could still access it if quadrant goes down there's it's engine the engineering side is really fun there's always decisions to be made and um but generally quadrants very flexible it makes these decisions easy. I see. Thank you. Bye. Sure. Thanks for coming. So we have probably about a minute and a half. It looks like a final question that may have actually already been answered. Good question, Giuseppe. No, Phi3 can extract the entities and relationships by itself, which is really cool. But this is a good point, and I'll try to wrap it up on this one. And this is where the ontology-free aspect is kind of controversial and useful for us, for our application, but maybe not so useful for others. In some cases, you actually want to limit the entities and you want to say, hey, I only want to extract relationships between people and organizations. I don't want anything else. And in that case, PHY3, the fine tune we made would not be very useful. It would be quite a bit problematic for you because you would end up getting entities that you don't want. And there are other models that have been fine tuned. I. I think it's called sci-fi or something. And it's really, and you can actually say, I only want these entities, or I only want these relationships. But no, Gliner is more or less separated completely from the phi3 component. Thank you, Rob. Thanks for joining us. How do people get a hold of you or what can they follow up on if they want to get more information? Sure. Head over to asknews.app and use it as your global news, please, if you'd like. We believe that algorithmic enforcement of diversity has a lot of power and is really useful when you're trying to properly understand what's going on in the world. So asknews.app is the best place to find us. From there, you can find all of our different products. We have an API. We have a New Plunker, which is an analyst tool that allows you to really visually interact with this information. So yeah, my name is Robert Kalk, C-A-U-L-K. You can find me on LinkedIn and send me a DM. And I'm always happy to geek out if there's ever anything. We have a Discord as well. You'll find that on asknews.app where we chat. I think I'm in the GraphGeeks Discord. So I'm happy to, to go chat over there as well, but thanks. I really appreciate everybody coming in and listening and, you know, we'll see where this world, where this kind of ecosystem evolves to. I think we're all sitting on the forefront wondering what's going to happen. And that's the funnest part of this. Thank you, Rob. Thank you everyone for joining. I'll see you all online and in discord and definitely follow up with questions if you have them thank you again Rob really appreciate it
GraphGeeks Explainer S2 Ep1: Exploring News at Scale with On-the-Fly Graphs
How do you build a knowledge graph that processes millions of new entities and relationships every day—and make it easy to explore through natural conversation? In this talk, Rob Caulk, the founder of Emergent Methods and open source veteran with over 1000 academic citations, will share how their team fine-tuned Phi-3-mini-4k to outperform Claude Sonnet 3.5 in graph extraction for dynamic knowledge wrangling at scale.

Rob will take you behind the scenes, breaking down key innovations: creative dataset engineering, novel loss metrics, and subjective evaluation methods that significantly boost accuracy in the AskNews software.

You’ll also learn how the team enabled on-the-fly sub-graph creation to enable targeted, domain-specific exploration of news. We’ll wrap up with a live demo, showing how to build and interact with a custom finance sub-graph—giving you practical strategies to apply knowledge graph exploration for your own use cases.

RESOURCES:
https://asknews.app/