A new artificial intelligence (AI) model developed by researchers at Aalto University and the Finnish Center for Artificial Intelligence (FCAI) is the first of its kind to predict how we move our eyes and fingers when typing on screens. The new development provides fresh insight into this difficult skill, which forces us to rely on our sense of sight to type and detect errors simultaneously.
The AI is capable of simulating how an individual would type a sentence on any keyboard design, and it both makes and detects errors. While the errors are not always caught right away, the AI is able to correct them just like us. Another aspect of the AI is that it can predict how peoples’ writing styles alter depending on the given circumstances, such as an auto-correct or different keyboard design.
Dr. Jussi Jokinen led the work on the new AI model, which was presented this month at ACM CHI on May 12.
“Previously, touchscreen typing has been understood mainly from the perspective of how our fingers move,” Jokinen says. “AI-based methods have helped shed new light on these movements: what we've discovered is the importance of deciding when and where to look. Now, we can make much better predictions on how people type on their phones or tablets.”
“Now that we have a realistic simulation of how humans type on touchscreens, it should be a lot easier to optimize keyboard designs for better typing — meaning less errors, faster typing, and, most importantly for me, less frustration,” Jokinen continues.
Accounting for Different Types of Users
The AI model doesn’t only work with one type of generic user, but it is able to adjust based on different types of users. This can include individuals with motor impairments, which the AI could assist by developing typing aids or specific interfaces.
Outside of medical issues, the AI can also account for individuals with different writing styles by doing things like identifying patterns when an individual makes repeated errors in texts. With this data, the best type of keyboard or auto-correct system for an individual can be determined.
The Team’s Previous Work
The researchers previously built their work around a cognitive model of the way humans type, which led to the creation of a generative model that was able to type by itself. All of this is part of the project of Interactive AI at the Finnish Center for Artificial Intelligence.
The research heavily relies on reinforcement learning, which works by creating a reward and/or punishment system, often used to teach robots how to solve issues through trial and error. Reinforcement learning was used by the team to simulate humans this time around, and it enabled them to generate behavior that is similar to humans’.
“We gave the model the same abilities and bounds that we, as humans, have. When we asked it to type efficiently, it figured out how to best use these abilities. The end result is very similar to how humans type, without having to teach the model with human data,”Jokinen says.
When compared to human typing data, the new model’s predictions were accurate. The team now aims to simulate slow and fast typing techniques, which could help in the design of learning modules for individuals looking to improve their typing skills.