Researchers Develop Tool Able to Turn Equations Into Illustrations

Researchers at Carnegie Mellon University have created a tool that is able to turn the abstractions of mathematics into illustrations and diagrams through software. 

The process works by users typing ordinary mathematical expressions which are then turned into illustrations by the software. One of the major developments in this project is that the expressions are not required to be basic functions, as in the case of a graphing calculator. Instead, they can be complex relationships coming from various different fields within mathematics. 

Penrose 

The tool was named Penrose by the researchers, inspired by the mathematician and physicist Roger Penrose, who is known for using complex mathematical and scientific ideas through diagrams and drawings. 

Penrose will be presented by researchers at the SIGGRAPH 2020 Conference on Computer Graphics and Interactive Techniques. The conference will take place virtually this year due to the COVID-19 pandemic. 

Keenan Crane is an assistant professor of computer science and robotics.

“Some mathematicians have a talent for drawing beautiful diagrams by hand, but they vanish as soon as the chalkboard is erased,” Crane said. “We want to make this expressive power available to anyone.”

Diagrams are not used as much in technical communication, due to the required amount of high-skill and tedious work required in order to produce them. To get around this, the Penrose tool allows experts to encode the steps in the system, and other users are then able to access this by using mathematical language. All of this means that the computer is doing most of the work. 

Katherine Ye is a Ph.D student in the Computer Science Department.

“We started off by asking: ‘How do people translate mathematical ideas into pictures in their head?'” said Ye. “The secret sauce of our system is to empower people to easily ‘explain' this translation process to the computer, so the computer can do all the hard work of actually making the picture.”

The Process

The computer first learns how the user wants the mathematical objects visualized, such as an arrow or a dot, and it then draws up multiple diagrams. The user selects and edits one of those diagrams. 

According to Crane, mathematicians should have no problem learning the special programming language that the team developed.

“Mathematicians can get very picky about notation,” he said. “We let them define whatever notation they want, so they can express themselves naturally.”

Penrose is seen as a step towards something even bigger. 

“Our vision is to be able to dust off an old math textbook from the library, drop it into the computer and get a beautifully illustrated book — that way more people understand,” Crane said.

The team that developed Penrose also included Nimo Ni and Jenna Wise, who are Ph.D. students in CMU’s Institute for Software Research (ISR); Jonathan Aldrich, professor in ISR; Joshua Sunshine, ISR senior research fellow; Max Krieger, cognitive science undergraduate; and Dor Ma’ayan, former master’s student at the Technion-Israel Institute of Technology. 

The research was supported by the National Science Foundation, Defense Advanced Research Projects Agency, the Sloan Foundation, Microsoft Research, and the Packard Foundation.