AI 101
What is NLU (Natural Language Understanding)?
Natural language understanding (NLU) is a technical concept within the larger topic of natural language processing. NLU is the process responsible for translating natural, human words into a format that a computer can interpret. Essentially, before a computer can process language data, it must understand the data.
Techniques for NLU include the use of common syntax and grammatical rules to enable a computer to understand the meaning and context of natural human language. The ultimate goal of these techniques is that a computer will come to have an “intuitive” understanding of language, able to write and understand language just the way a human does, without constantly referring to the definitions of words.
Defining NLU (Natural Language Understanding)
There are numerous techniques that computer scientists and NLP experts use to enable computers to understand human language. Most of the techniques fall into the category of “syntactic analysis”. Syntactic analytic techniques include:
- lemmatization
- stemming
- word segmentation
- parsing
- morphological segmentation
- sentence breaking
- part of speech tagging
These syntactic analytic techniques apply grammatical rules to groups of words and attempt to use these rules to derive meaning. In contrast, NLU operates by using “semantic analysis” techniques.
Semantic analysis applies computer algorithms to text, attempting to understand the meaning of words in their natural context, instead of relying on rules-based approaches. The grammatical correctness/incorrectness of a phrase doesn’t necessarily correlate with the validity of a phrase. There can be phrases that are grammatically correct yet meaningless, and phrases that are grammatically incorrect yet have meaning. In order to distinguish the most meaningful aspects of words, NLU applies a variety of techniques intended to pick up on the meaning of a group of words with less reliance on grammatical structure and rules.
NLU is an evolving and changing field, and its considered one of the hard problems of AI. Various techniques and tools are being developed to give machines an understanding of human language. Most NLU systems have certain core components in common. A lexicon for the language is required, as is some type of text parser and grammar rules to guide the creation of text representations. The system also requires a theory of semantics to enable comprehension of the representations. There are various semantic theories used to interpret language, like stochastic semantic analysis or naive semantics.
Common NLU techniques include:
Named Entity Recognition is the process of recognizing “named entities”, which are people, and important places/things. Named Entity Recognition operates by distinguishing fundamental concepts and references in a body of text, identifying named entities and placing them in categories like locations, dates, organizations, people, works, etc. Supervised models based on grammar rules are typically used to carry out NER tasks.
Word-Sense Disambiguation is the process of determining the meaning, or sense, of a word based on the context that the word appears in. Word sense disambiguation often makes use of part of speech taggers in order to contextualize the target word. Supervised methods of word-sense disambiguation include the user of support vector machines and memory-based learning. However, most word sense disambiguation models are semi-supervised models that employ both labeled and unlabeled data.
Examples of NLU (Natural Language Understanding)
Common examples of NLU include Automated Reasoning, Automatic Ticket Routing, Machine Translation, and Question Answering.
Automated Reasoning
Automated reasoning is a discipline that aims to give machines are given a type of logic or reasoning. It’s a branch of cognitive science that endeavors to make deductions based on medical diagnoses or programmatically/automatically solve mathematical theorems. NLU is used to help collect and analyze information and generate conclusions based off the information.
Automatic Ticket Routing
NLU is often used to automate customer service tasks. When a customer service ticket is generated, chatbots and other machines can interpret the basic nature of the customer’s need and rout them to the correct department. Companies receive thousands of requests for support every day, so NLU algorithms are useful in prioritizing tickets and enabling support agents to handle them in more efficient ways.
Machine Translation
It’s difficult to accurately translate speech or text from one language to another language. In fact, machine translation is one of the most difficult problems in NLP and NLU. Many machine translation systems rely on linguistic rules to translate between languages, but researchers are pursuing more sophisticated ways of translating between languages. NLU machine translation attempts to enable more accuracy translation by preserving the context and semantic information associated with the target text. The most accurate machine translation systems combine linguistic rules with algorithms that extract semantic meaning.
Question Answering
Speech recognition uses NLU techniques to let computers understand questions posed with natural language. NLU is used to give the users of the device a response in their natural language, instead of providing them a list of possible answers. When you ask a digital assistant a question, NLU is used to help the machines understand the questions, selecting the most appropriate answers based on features like recognized entities and the context of previous statements.
