The concept tree, the QDEX, and SemanticRank
The sentence-level analysis of Internet information is the key to Hakia's engine. The process begins by breaking down each sentence into "knowledge bits" and then applying a proprietary concept tree to the entire sentence with the goal of understanding what the words mean and how they fit together.
Just like a map of the US shows the links between cities and towns, a concept tree shows the links between ideas. Every concept in the tree has a parent, children, and siblings, making this tree the equivalent of a California redwood. Berkan calls the process of creating the tree "almost like rewriting Webster's dictionary." It's painstaking work, and "hundreds of linguists" have worked on it for the last few years.
This tree is applied to each sentence, helping the engine to understand the event, the agent, and the theme of the sentence, which provides a basic understanding of what was written. All of this processing falls under the heading of "ontological semantics," or "OntoSem" in Hakia-speak (for more information, take a look at Dr. Victor Raskin's page on the subject; Raskin is currently an adviser to Hakia).
Part of the concept tree for "bow"
Those curious about how concept trees work in Hakia's system can see an OntoSem example using the word "bow." The word is parsed into its various meanings (a weapon, part of a musical instrument, etc.), and each is broken down into various syntactic and semantic structures that show how the word might operate in sentences and how it connects to other concepts. Imagine doing this for the 100,000+ entries in Hakia's dictionary, and it's easy to see why it has taken several years to develop.
Web pages are analyzed using OntoSem and the results are passed to the QDEX algorithms. QDEX replaces the typical "index" used by many other search engines; instead of keywords, it stores knowledge bits. Berkan describes QDEX as the "middleware that enables full-scale semantic analysis," and the software goes to work on web pages gathered by Hakia's spidering system. Once the text in these pages has gone through OntoSem processing and the words and concepts are all tagged, QDEX algorithms skim information and attempt to generate "all possible queries that can be asked to this content."
Even for a simple web page, this could obviously result in millions of different queries based on the listed concepts. But humans reading the page would be likely to ask only a few questions about the information it contains, and one of the main jobs of QDEX is to reduce the vast number of possible queries into the much smaller number of likely human queries. QDEX then stores these possible queries that point back to the page or sentence where the answer can be found. All of this work is done off-line, of course; it takes far too much processing power for real-time use on vast amounts of data.
On its web site, Hakia describes this technology as providing "great flexibility in a search engine platform for utilizing semantically rich data and multiple-thread processing of equivalent queries. Otherwise, deep semantic analysis is virtually impossible over a vast amount of textual data area."
With the QDEX in place, the search engine is ready to tell users about Richard Nixon, or puppy dogs. The QDEX has already generated all of the necessary queries when it analyzed the webpage, but these now need to be ranked for display. Hakia uses a final technology called SemanticRank to comb through the stored queries and decide which ones best fit the search terms (which have themselves been parsed with OntoSem technology). Hakia claims that "no keyword matching or Boolean algebra [is] involved" in the process; it's all about concept-matching.
Well, mostly. SemanticRank does take credibility and age of information into account, though the company stresses that it does not do this by counting link referrals (popularity). Berkan tells Ars that a true semantic search should never require page rank; it should be about the quality of the information. Hakia evaluates credibility by analyzing the way a page is written (does it use proper English, for instance), who authored it, and what site it's on. Medical claims on blogs will never make it above information from the NIH, Berkan says, no matter how popular they are, because the system understands that the NIH is a credible institution.