During the pandemic, interest in natural language processing (NLP) technologies exploded as enterprises looked to augment their human workforces with AI-powered systems. Applications like neural machine translation, chatbots, hiring tools, and conversational search have made the business value of NLP more apparent, especially as companies embark on digital transformations. KPMG reports that the pandemic has resulted in an acceleration of digital transformation by “months or even years” across some industries.

Advisory firm Mordor Intelligence forecasts the NLP market will more than triple its 2019 revenue of $6.94 billion by 2025, reflecting a potential to enable natural conversations, more efficient operations, reduced costs, higher customer satisfaction, and improved analysis. But while NLP is becoming increasingly important, it remains a difficult field to parse, in part because it encompasses a large number of subfields.

What is NLP?

NLP is itself a subfield of linguistics, computer science, and AI concerned with the interactions between machines and the human language. NLP systems can “understand” to some degree the contents of documents, including the contextual nuances of the words within them.

In the early days, many NLP systems relied on hand-coded sets of rules called symbolic methods in order to parse text files. But beginning in the 2010s, machine learning approaches became widespread, like word embeddings, which are representations of words that capture the semantic properties of those words.

These modern systems have advantages over the rule-dependent software of yore. For example, they focus on common cases by nature of the AI model training process and make use of statistical inference to handle unfamiliar words. Moreover, while systems based on rules can only be made more accurate by increasing rule complexity, AI-based systems’ accuracy generally corresponds with the amount of training data.

NLP drives software that responds to voice commands and summarizes enormous volumes of text, and it’s a core part of speech tagging, the process of determining the part of speech of a particular word based on its use. NLP is also important in sentiment analysis, which attempts to extract subjective qualities including attitudes, emotions, sarcasm, confusion, and suspicion from text, as well as identifying words or phrases as potentially useful entities.

Unlocking data value

Accenture notes that businesses can leverage NLP in two primary ways: query understanding and content understanding. When deployed strategically, NLP can provide better, more targeted customer and employee service responses by understanding questions and intent, for example. It can pull out entities from documents to spotlight any relevant products, processes, and procedures and identify and comprehend the meaning of natural language content, including reports and emails, to provide answers in plain English.

NLP is increasingly being used in cognitive search, a type of enterprise search technology that uses AI to return relevant info to users via apps, APIs, and desktop operating systems. Platforms like Microsoft’s Project Cortex, Amazon Kendra, and Google Cloud Search tap NLP to understand not only document minutia but the searches that employees across an organization might pose, like “How do I invest in our company’s 401k?” versus “What are the best options for my 401k plan?”

Another popular use of NLP is spam detection, where the technology is applied to scan emails for language that might indicate malware or phishing attempts. Indicators that NLP systems can be tuned to detect include overused financial terms, characteristic bad grammar, threatening language, inappropriate urgency, misspelled company names, and more. As of February 2019, Google was using NLP and other AI systems to block 100 million additional spam messages every day. And startups including Armorblox employ NLP to analyze sensitive information in emails and documents and to protect against data- and identity-related attacks.

Some startups are applying NLP to spot bugs in user reports, like UniQ. There’s also companies like Chorus.ai, Observe.ai, Amenity Analytics, and Cogito, which analyze call center and sales call data using NLP. Amenity Analytics language understanding systems parse regulatory filings and earnings calls for key points. And Klevu personalizes ecommerce search with NLP techniques.

NLP-powered voice assistants and chatbots have seen an uptick in usage lately. That’s because they enable brands to tailor offers and recommendations without humans in the loop. Both chatbots and assistants leverage customer, product, and interaction data to improve experiences in real time, leading to reduced wait times, service costs, and customer churn. And they can have value beyond customer service. For example, NLP-powered tools can assist in the employee onboarding process, fielding screening questions, recording answers, and guiding new employees through company policies and protocols.

Chatbots and voice assistants dovetail with NLP-powered document processing systems like Google’s DocAI platform, which processes loan applicants’ asset documents in addition to invoices, receipts, and more. The business value of document processing is nothing to scoff at — companies spend an average of $20 to file and store a single document, by some estimates, and only 18% of companies consider themselves paperless.

Challenges and deployment

Like any technology, NLP has its flaws. Improperly tuned models risk reinforcing undesirable stereotypes, particularly if the training data is commonly sourced from communities with prejudices around gender, race, and religion. One solution to models’ shortcomings might be developing tools for customers to evaluate quality. Several already exist, like Robustness Gym, a framework developed by Salesforce’s natural language processing group that aims to unify the patchwork of existing robustness libraries to accelerate the development of novel natural language model testing strategies.

Salesforce conversational design principal Greg Bennett also advocates including stakeholders throughout the NLP system design process so biases can be accounted for and mitigated — at least to the extent possible. “Any institution has the opportunity to use [an NLP system] to essentially extend itself in a relationship with a customer — with prospective students, with job applicants, the list goes on. These are opportunities to create relationships and have a meaningful exchange,” he told VentureBeat in a recent interview.

For enterprises considering deploying an NLP system, it’s best to start by identifying the concrete business problems it might solve. Next should come an evaluation of in-house versus external vendor solutions. Once the solutions are settled upon, development, testing, and deployment can begin in earnest.

As Accenture writes, NLP can pay dividends. That’s perhaps why NLP budgets in the enterprise increased by 10% to 30% in 2020 compared with 2019, according to Markets and Markets — despite the fact that IT spending as a whole decreased significantly during the early days of the pandemic.

“NLP has become an essential enabler of the AI evolution in today’s enterprises … With a well-implemented NLP solution in place, organizations can enable a deeper understanding of unstructured content, providing enhanced business intelligence and analytics,” Accenture wrote. “Analyzing structured data alone is no longer enough. Sophisticated business analyses, predictions, and decision making all need more.”