Case Study: Machine Learning vs. Natural Language Processing

Why you shouldn't use Machine Learning as a substitute for real NLP.

There are 2 kinds of Natural Language Processing…

Today, industry-leading NLP is built on AI that detects patterns in data that can then be leveraged in understanding user inputs. This creates an approach that is flexible in adapting to the nuances and ambiguity of languages while boosting accuracy between results and user intents.

However, there still exists implementations of an old-school approach to NLP that relies on machine learning algorithms and predetermined rules. In this approach, algorithms are fed what words and phrases to detect in order to return correlating responses — this imprecise method of “understanding” language results in limited accuracy.

With the latest strides in technology, there’s no acceptable excuse to continue using basic machine learning and keyword-driven algorithms as a substitute for modern, sophisticated natural language processing.

We put our NLP to the test.

In a short comparative exercise, we put another NLP-driven chatbot against Inbenta’s own to examine their real responses to “natural” language. The following questions were posed both to the Inbenta Chatbot and to another popular chatbot service on the market that advertises its use of NLP. For the sake of this comparison, we’ll refer to it as the anonyBot.

Without making any changes to our lexicon, we took the FAQs used by anonyBot and plugged them into our Inbenta knowledge base.

The results of the experiment pointed out a clear discrepancy between the two conceptions of Natural Language Processing.

To further demonstrate one of the many steps in our NLP process, we’ve tagged each result shown here with its semantic score — a calculated percentage of how close of a match the ending result is to the user’s original query.

How semantic scoring works:

Using Inbenta’s NLP stack, we’re able to abstract a query down to clusters of lexical units, functions, and concepts that represent meaning, or intent, rather than language. We then use an algorithm that ranks existing contents to the intent, and the strongest match is returned to the customer in the chatbot’s response.


The Experiment:

Question 1: is there a limit of the size of db?

anonyBot:No answer [✗]

Inbenta: How large a knowledge base can I create? [✓]

Our semantic score: 58.8%

Question 2: which formats do you support?

anonyBot: No answer [✗]


Inbenta: What format does the tool expect the file content to be? [✓]

Our semantic score: 80.9%

Question 3: I forgot my password

anonyBot: No answer [✗]

Inbenta: How do I login? [✓]

Our semantic score: 57.1%

Question 4: I made i change but I can’t see it yet

anonyBot: No answer [✗]

Inbenta: The updates I made to my knowledge base are not reflected on publish. Why not? [✓]

Our semantic score: 58.4%

Question 5: is my data secure?

anonyBot: No answer [✗]

Inbenta: How safe is my knowledge base data? [✓]

Our semantic score: 92.4%


To learn more about the sophistication behind real natural language processing, check out this high-level summary delivered by Inbenta Co-founder and CEO, Jordi Torras.

 

Inbenta is a leader in natural language processing and artificial intelligence for customer support, e-commerce and conversational chatbots, providing an easy-to-deploy solution that improves customer satisfaction, reduces support costs, and increases revenue.

Interested in finding out more? Our team of experts is available to show you how Inbenta can benefit your company

 

Let’s get in touch