natural language search

Keyword-based search versus natural language search

Searching is the starting point to the world's information, yet many websites still tend to rely on clunky navigation. Users are becoming more and more impatient when they visit websites and can’t find the desired information in an immediate way. The cost of not enabling natural language search is huge for website owners.

Just consider these few facts:

  • When attempting to find the target information through site navigation fails, 50% of users will turn to search
  • In e-commerce websites, during a study of more than 2,000 shoppers, 71% used the search engine to find their products
  • In customer service, 90% of consumers who have made an online purchase said they used site search to access self-service content

A survey conducted by Kelton Research on the current state of search showed that 1001 adults felt that, unlike natural language search, keyword search creates more time spent, resulting in “search engine fatigue, frustration and the desire to want the search engine to read their minds”.

  • 65.4% of Americans say they’ve spent two or more hours in a single sitting searching for specific information by means of a search engine.
  • 72.3% experience “search engine fatigue” when researching a topic on websites.
  • 75.1% of those who experience search engine fatigue report getting up and physically leaving their computer without the information they were seeking.
  • 78% “wished” that search engines could actually kind of “read their minds” to produce the results they were looking for.

Why is search so widely used yet creates such a negative impact on user’s opinions?

The problem is that the most common search engines are “keyword based,” which means all search queries and retrievals will operate under the keyword rules of stemming.

Increasing complications further, many text indexing systems generally pick up every word in the text except commonly occurring stop words such as “a,” “an,” “the,” “is,” “and,” “or,” and “www”. This means these search engines are completely devoid of offering “meaning,” which is why users are so frustrated. Users want answers instantly by way of asking naturally, not by guessing with a single or group of words that a search engine must logically pair with potential words it “might” be related too. There is a problem with keyword search and companies are now seeing the pain users have been going through over the years.


What is the problem with keyword searching?

Keyword searches have a tough time distinguishing between words that are spelled the same way but mean something different (i.e. hard cider, a hard stone, a hard exam, and the hard drive on your computer). This often results in hits that are completely irrelevant to your query.

Most sites offer two different types of searches–“basic” and “refined” (also known as “advanced”). In a “basic” search, you just enter a keyword without sifting through any pulldown menus of additional options. Depending on the engine, though, “basic” searches can be quite complex.

Unless they are a search guru, the typical average Joe is often not going to use the advanced search. The reality is that these “advanced” search tools are rarely implemented and tend to be useless because it still applies the same keyword pairing principles. So you get more refined bad results, not more accurate results.


Natural language search solved the keyword dilemma  

Unlike keyword search systems, natural language search systems focus on meaning and the natural way humans ask and offer answers to each other, not just what you say in a few words. Natural language is concept-based, which means it returns search hits on documents that are “about” the subject/theme you’re exploring, even if the words in the document don’t match at all the words you enter into the query.


… but, what is natural language?

Natural language is what we use every day as a means of communication with other humans. It has a syntax and complies with principles of economy and optimality. One of the biggest challenges in computer science today is the creation of a computer system that is able to understand our natural language.

Inbenta applies NLP technology and the meaning-text theory to create software that understands natural language and implements a truly “intelligent” search experience for your website, using full statements instead of mere keywords.

Natural language is the closest a search engine can get to “reading the minds” of internet users. By letting users express themselves in their own words, as they would when addressing a real human being, they will have the ability to use technology that understands what they are actually looking for rather than acting as a guessing machine. With natural language search, we can relieve user’s search engine fatigue and turn their search experience into an effective, positive, more human experience.

Inbenta Team
by Inbenta Team