Baidu, one of the biggest tech companies in China, has recently developed a new method of teaching AIs to understand language. As reported by TechnologyReview, the company recently beat out Microsoft and Google at the General Language and Understanding Evaluation (GLUE) competition, achieving state of the art results.
GLUE is comprised of nine different tests, with each test measuring a different task important to the understanding of language, such as discerning names of entities in a sentence and discerning in what context the pronoun “it” is being used when there are numerous potential candidates. The average human typically scores around 87 points on GLUE, out of a possible 100. Baidu’s new model, ERNIE, cracked the 90 point threshold.
Researchers are always trying to improve the performance of their models at GLUE, and therefore the current standard set by Baidu will probably be outdone soon. However, what makes Baidu’s achievements notable is that the learning approach they use seems to be able to generalize to other languages. Even though the model was developed to interpret Chinese, the same principles make it better and interpreting the English language. ERNIE stands for “Enhanced Representation through knowledge Integration”, and it follows the development of the BERT (“Bidirectional Encoder Representations from Transformers”) language model.
BERT set a new standard for language understanding due to the fact that it was a bidirectional model. Previous language models were only capable of interpreting data that flowed in one “direction”, looking at a word that came either before or after the target word as context. BERT was able to able to implement a bidirectional approach that could use both previous and later words in a sentence to help figure out the meaning of a target word. BERT uses a technique called masking to make bidirectional analysis possible, choosing a word in a sentence and hiding it, which splits up the possible context for that word in preceding and succeeding context clues.
In the English language, the word is the dominant semantic unit, people look at whole words rather than individual characters to discern meaning. It’s possible to remove a word from its context and still have that word maintain its meaning, and the meaning of individual characters is almost always the same. In contrast, the Chinese language relies much more on how characters are matched together with other characters when discerning meaning. Characters can mean different things depending on the characters around them.
The Baidu research team essentially took the model BERT used and expanded it, hiding strings of characters instead of full words. The AI system was also trained to differentiate between random strings and meaningful strings in order that the right strings of characters could be masked. This makes ERNIE proficient at retrieving information from a text document and carrying out machine translation. The research team also found out that their training method also resulted in a model that could distinguish English phrases better than many other models could. This is because English sometimes, although rarely, uses word combinations that express different meanings when they are joined together versus when they are by themselves. Proper names and idioms or colloquialisms, such as “chip off the old block” are examples of such linguistic phenomena.
ERNIE makes use of multiple other training techniques in order to optimize performance, including analyzing sentence order and distance when interpreting paragraphs. A continuous training method is also used, which allows ERNIE to train on new data and learn new patterns without forgetting previously acquired knowledge.
Baidu currently uses ERNIE to enhance the quality of search results. ERNIE’s latest architecture will be detailed in an upcoming paper to be presented at the 2020 Association for the Advancement of Artificial Intelligence conference.
Alexander Hudek, Co-Founder & CTO of Kira Systems – Interview Series
Alex Hudek is the Co-Founder & CTO of Kira Systems. He holds Ph.D and M.Math degrees in Computer Science from the University of Waterloo, and a B.Sc. from the University of Toronto in Physics and Computer Science.
His past research in the field of bioinformatics focused on finding similarities between DNA sequences. He has also worked in the areas of proof systems and database query compilation.
When did you initially become interested in machine learning and AI?
I’ve always been interested in computer science. In undergrad I took courses in algorithms for planning and logic, machine learning and AI, numerical computing, and other topics. My interest in machine learning grew more specifically during my PhD at the University of Waterloo. There, I used machine learning methods to study DNA. Afterwards, I dove more deeply into formal logics as part of my postdoctoral research. Logic and reasoning is in some ways the “other side” of the coin in approaches to AI and I felt it important to know more about it.
Some of your past research in the field of bioinformatics focused on finding similarities between DNA sequences. Could you discuss some of this work?
The main body of my thesis involved building a more realistic model DNA mutation using Hidden Markov Models. I used this more complex model in a new algorithm designed to find regions of DNA that share common ancestry with other species. In particular, this new algorithm can find much more weakly related sequence regions than previous algorithms for the task.
Before my PhD, I worked in a research lab that was part of the human genome project. One of the most notable projects I helped complete was the first complete draft of human chromosome 7.
What was the initial inspiration behind launching Kira?
The idea for Kira came from my co-founder, Noah Waisberg. He had spent hours in his career as a lawyer doing the sort of work we’ve now built AI to do. It was an interesting idea to me because it involved natural language and the problem was well scoped, and I could see the business potential. There is something alluring about building AI that can understand human language because language is so closely related to human cognition.
Can you describe what Contract Analysis Software is and how it benefits legal professionals?
Kira uses supervised machine learning, meaning an experienced lawyer feeds provisions from real contracts into a system designed to learn from those examples. The system studies this data, learns what language is relevant, and builds probabilistic provision models. The models are then tested against a set of annotated agreements that the system is unfamiliar with in order to determine its readiness. This highly accurate machine learning technology can identify and analyze virtually any provision in any contract, resulting in customer-reported time savings of 20-90%. This increased productivity helps Law Firms by increasing their Realization Rates, gives them more opportunity to grow their revenue and preserve their existing clients. For corporations, it drives better productivity in-house reducing the amount of external legal spend required.
Natural Language Processing (NLP) is difficult for most companies, could you discuss some of the additional challenges that are faced when it comes to processing legal terminology and other nuances that are unique to the legal profession?
For many people legal language can seem very foreign, but it turns out that from a machine learning perspective it’s not actually that different. There are a few more unique things; capitalization is more important and sentences can be much longer than normal, but overall we haven’t needed significantly different NLP approaches than in other domains.
One aspect that is significantly different is the need for data privacy and customization. Legal professionals are required to keep client data confidential, and using it in a machine learning product that pools or shares training data is at odds with those requirements. In fact, even keeping training data is often not possible as they have obligations to delete client data after a project concludes. Thus, being able to train models without vendors in the loop becomes critical, as do machine learning techniques that make it hard or impossible to recover any part of the training data by inspecting learned models. Techniques that allow you to take an existing model and update it with new training data without retraining from scratch are also a must have.
On the customization front, there is a need for clients to be able to build their own models. This is because for more complex legal concepts there can be reasonable disagreement among professionals, and firms often want to tune or build models to match their own unique positions.
Could you describe how deep learning is used to categorize data within Kira software?
We don’t use much deep learning in our product, though our internal research team does spend a lot of time evaluating and exploring deep learning solutions. So far, on the sorts of problems we face, deep learning techniques are only matching non-deep learning approaches, or at best getting a very small increase. Given the huge computation overhead of deep learning methodologies, as well as challenges in keeping training data private, they haven’t been compelling enough to adopt so far.
That said, we do find deep learning approaches to be very compelling and we think they have a potential to become big in NLP one day. To that end, we continually evaluate and explore deep learning NLP approaches so that we can be ready to adopt when the advantages start outweighing the disadvantages.
What are some of the built-in provision models that Kira offers?
Currently Kira can identify and extract over 1,000 built-in provisions, clauses, and data points (smart fields). They relate to a multitude of different topics, from M&A Due Diligence—which Kira was originally conceived to assist with—to Brexit; to Real Estate. The smart fields are built by our team of subject matter experts that include experienced lawyers and accountants. With our machine learning technology, Kira’s standards require virtually every smart field to achieve a minimum of 90% recall, meaning our software will find 90% or more of the provision, clause or data point you’re specifically looking for within your contracts or documents, reducing risks and errors in the contract review process. In addition, an unlimited number of custom fields can be created/taught by a firm to automatically identify and extract relevant insights using our Quick Study tool.
The legal world is often known for being slow to adopt new technology. Do you find that there’s an education hurdle when it comes to educating law firms?
Lawyers really like to know how things work, so education is important. It’s no harder to teach lawyers about machine learning and AI then other professionals, but it is definitely required to have training materials ready. Many of the adoption hurdles are social too; people often ask about best practices in adapting their internal processes to use AI, or are interested in how they can use AI to change their business offerings in a way that gives them advantages beyond just efficiency improvements.
Compared to when we started Kira Systems in 2011, law firms today are far more savvy about AI and technology. Many have innovation teams who are tasked with investigating new technology and encouraging adoption of new solutions.
Is there anything else that you would like to share about Kira?
Academic literature and open source machine learning libraries were instrumental in helping us bootstrap the company. We believe that open information and software is a huge boon to the world. In light of that, I’m especially happy that our research team publishes the results of many of our research efforts in academic journals and conferences. Aside from demonstrating that we push the boundaries of the state of the art, this allows us to give back to the communities that helped us get started, and that we continue to get a ton of value from. You can find our papers at https://kirasystems.com/science/.
To learn more visit Kira Systems.
DeepScribe AI Can Help Translate Ancient Tablets
Researchers from the University of Chicago’s Oriental Institute and the Department of Computer Science have collaborated to design an AI that can help decode tablets from ancient civilizations. According to Phys.org, the AI is called DeepScribe and was trained on over 6,000 annotated images pulled from the Persepolis Fortification Archive, when it is complete the AI model will be able to interpret unanalyzed tablets, making studying ancient documents easier.
Experts who study ancient documents, like the researchers who are studying the documents created during the Achaemenid Empire in Persia, need to translate ancient documents by hand, a long process that is prone to errors. Researchers have been using computers to assist in interpreting ancient documents since the 1990s, but the computer programs that were used were of limited help. The complex cuneiform characters, as well as the three-dimensional shape of the tablets, put a cap on how useful the computer programs could be.
Computer vision algorithms and deep learning architectures have brought new possibilities to this field. Sanjay Krishnan, from the Department of Computer Science at OI collaborated with associate professor of Assyriology Susanne Paulus to launch the DeepScribe program. The researchers oversaw a database management platform called OCHRE, which organized data from archaeological excavations. The goal is to create an AI tool that is both extensive and flexible, able to interpret scripts from digfferent geographical regions and time periods.
As Phys.org reported, Krishnan explained that the challenges of recognizing script, which archaeological researchers face, are essentially the same challenges faced by computer vision researchers:
“From the computer vision perspective, it’s really interesting because these are the same challenges that we face. Computer vision over the last five years has improved so significantly; ten years ago, this would have been hand wavy, we wouldn’t have gotten this far. It’s a good machine learning problem, because the accuracy is objective here, we have a labeled training set and we understand the script pretty well and that helps us. It’s not a completely unknown problem.”
The training set in question is the result of taking the tablets and translations, from over approximately 80 years of the archaeological research done at OI and U Chicago and making high-resolution annotated images from them. Currently, the training data is approximately 60 terabytes in size. Researchers were able to use the dataset and create a dictionary of over 100,000 individually identified signs that the model could learn from. When the trained model was tested on an unseen image set, the model achieved approximately 80% accuracy.
While the team of researchers is attempting to increase the accuracy of the model, even 80% accuracy can assist in the process of transcription. According to Paulus, the model could be used to identify or translate highly repetitive parts of the documents, letting experts spend their time interpreting the more difficult parts of the document. Even if the model can’t say with certainty what a symbol translates to, it can give researchers probabilities, which already puts them ahead.
The team is also aiming to make DeepScribe a tool that other archeologists can use in their projects. For instance, the model could be retrained on other cuneiform languages, or the model could make informed estimates about the text on damaged or incomplete tablets. A sufficiently robust model could potentially even estimate the age and origin of tablets or other artifacts, something typically done with chemical testing.
The DeepScribe project is funded by the Centre for the Development of Advanced Computing (CDAC). Computer vision has been used in other CDAC-funded projects as well, like a project intended to recognize style in works of art and a project designed to quantify biodiversity in marine bivalves. The team of researchers is also hoping their collaboration will lead to future collaborations between the Department of Computer Science and OI at the University of Chicago.
AI Trained To Be A Dungeon Master And Generate Plots For Dungeons And Dragons
Artificial intelligence has mastered even extremely complex games like chess and Go. However, these games have pre-defined rules and very specific methods of interaction that don’t lend themselves to creative choices. A role-playing game like Dungeons and Dragons (DnD) has infinitely more ways to play than a game of chess does, but this hasn’t stopped researchers from trying to develop AI systems capable of improvising storylines for DnD or similar tabletop role-playing games.
AI researchers are constantly working on new ways to improve the generative language abilities of AI. One of the biggest advances in the past couple of years is the development GPT-2, which was able to generate coherent stories on the fly. However, as Wired reported, Georgia Tech graduate student Lara Martin conceived of using DnD as a test case for an AI’s generative language ability. The goal is essentially to create an AI dungeon-master, capable of creating new scenarios for the game and adapting these scenarios.
According to Wired, Martin has been working on the AI dungeon master since 2018. Language generation models often use either rules-based approaches or neural networks based approaches. Recently, there has been an increasing interest in combining the two techniques to generate language. Martin’s approach utilizes rules-based language generation strategies alongside deep neural networks. Martin’s approach to language generation relies on the idea of “events”. Events consist of various parts of speech like objects, subjects, and verbs, which the model combines into event objects that are coherent. The model was trained on storylines from popular sci-fi Tv shows like Futurama and Doctor Who. The model is primed with a string of text, which it will analyze for events. After extracting the events from the priming text, it will attempt to continue the plot by generating new events. Martin was able to expand on this basic approach and guide the model towards generating certain desired events, like the marriage of two characters within the story.
Martin isn’t the only researcher attempting to design AI’s capable of telling stories. For example, machine learning researcher Nick Walton recently developed AI Dungeon, which makes use of GPT-2 models to create an AI-generated text-adventure game. While AI Dungeon typically renders text that is at least coherent, it tends to lose track of the overall narrative, start strange new plot threads, and generally behave oddly to player input. Despite these limitations, the game has proved rather popular, with over a million people playing it.
Martin acknowledges the limits of the model, stating that the model often gets confused, generating plot events that don’t make logical sense, and that “we’re nowhere close to this being a reality yet”. Despite this, Martin still hopes that the model will lead to something useful in the future. Martin is also hopeful that the project could potentially give us insight into how the creation of stories leverages different aspects of intelligence like imagination and embodiment.
“If we could create a convincing AI DM, it would tell us more about how we create and experience these worlds,” Martin explained to Wired.
It could also be argued that the challenge of accomplishing a feat as difficult as creating a dungeon master AI is reason enough to pursue the project. Noah Smith, an AI and language professor at the University of Washington explained that large goals sometimes help create usable applications, even if the challenge itself isn’t accomplished in a timely fashion.
“Sometimes grand challenge goals are helpful in getting a lot of researchers moving in a single direction. And some of what spins out is also useful in more practical applications.”
- Neural Hardware and Image Recognition
- Charles J. Simon, Author, Will Computers Revolt? – Interview Series
- AI Makes it Easier for Drones to Scan and Excavate Terrain
- Noah Schwartz, Co-Founder & CEO of Quorum – Interview Series
- Anton Dolgikh, Head of AI, Healthcare and Life Sciences at DataArt – Interview Series