Andrew Stein, Software Engineer Waymo – Interview Series

Andrew Stein is a Software Engineer who leads the perception team for Waymo Via, Waymo's autonomous delivery efforts. Waymo is an autonomous driving technology development company that is a subsidiary of Alphabet Inc, the parent company of Google.

What initially attracted you to AI and robotics?

I always liked making things that “did something” ever since I was very young. Arts and crafts could be fun, but my biggest passion was working on creations that were also functional in some way. My favorite parts of Mister Rogers’ Neighborhood were the footage of conveyor belts and actuators in automated factories, seeing bottles and other products filled or assembled, labeled, and transported. I was a huge fan of Legos and other building toys. Then, thanks to some success in Computer Aided Design (CAD) competitions through the Technology Student Association in middle and high school, I ended up landing an after-school job doing CAD for a tiny startup company, Clipper Manufacturing. There, I was designing factory layouts for an enormous robotic sorter and associated conveyor equipment for laundering and organizing hangered uniforms for the retail garment industry. From there, it was off to Georgia Tech to study in electrical engineering, where I participated in the IEEE Robotics Club and took some classes in Computer Vision. Those eventually led me to the Robotics Institute at Carnegie Mellon University for my PhD. Many of my fellow graduate students from CMU have been close colleagues ever since, both at Anki and now at Waymo.

You previously worked as a lead engineer at Anki a robotics startup. What are some of the projects that you had the opportunity to work on at Anki?

I was the first full-time hire on the Cozmo project at Anki, where I had the privilege of starting the code repository from scratch and saw the product through to over one million cute, lifelike robots shipped into people's homes. That work transitioned into our next product, Vector, which was another, more advanced and self-contained version of Cozmo. I got to work on many parts of those products, but was primarily responsible for computer vision for face detection, face recognition, 3D pose estimation, localization, and other aspects of perception. I also ported TensorFlow Lite to run on Vector's embedded OS and helped deploy deep learning models to run onboard the robot for hand and person detection.

I also built Cozmo's and Vector's eye rendering systems, which gave me the chance to work particularly closely with much of Anki’s very talented and creative animation team, which was also a lot of fun.

In 2019, Waymo hired you and twelve other robotics experts from Anki to adapt its self-driving technology to other platforms, including commercial trucks. What was your initial reaction to the prospect of working at Waymo?

I knew many current and past engineers at Waymo and certainly was aware of the company's reputation as a leader in the field of autonomous vehicles. I very much enjoyed the creativity of working on toys and educational products for kids at Anki, but I was also excited to join a larger company working in such an impactful space for society, to see how software development and safety are done at this organizational scale and level of technical complexity.

Can you discuss what a day working at Waymo is like for you?

Most of my role is currently focused on guiding and growing my team as we identify and solve trucking-specific challenges in close collaboration with other engineering teams at Waymo. That means my days are spent meeting with my team, other technical leads, and product and program managers as we plan for technical and organizational approaches to develop and deploy our self-driving system, called the Waymo Driver, and extend its capabilities to our growing fleet of trucks. Besides that, given that we are actively hiring, I also spend significant time interviewing candidates.

What are some of the unique computer vision and AI challenges that are faced with autonomous trucks compared to autonomous vehicles?

While we utilize the same core technology stack across all of our vehicles, there are some new considerations specific to trucking that we have to take into account. First and foremost, the domain is different: compared to passenger cars, trucks spend a lot more time on freeways, which are higher-speed environments. Due to a lot more mass, trucks are slower to accelerate and brake than cars, which means the Waymo Driver needs to perceive things from very far away. Furthermore, freeway construction uses different markers and signage and can even involve median crossovers to the “wrong” side of the road; there are freeway-specific laws like moving over for vehicles stopped on shoulders; and there can be many lanes of jammed traffic to navigate. Having a potentially larger blind spot caused by a trailer is another challenge we need to overcome.

Waymo’s recently began testing a driverless fleet of heavy-duty trucks in Texas with trained drivers on-board. At this point in the game, what are some of the things that Waymo hopes to learn from these tests?

Our trucks test in the areas in which we operate (AZ / CA / TX / NM) to gain meaningful experience and data in all different types of situations we might encounter driving on the freeway. This process exercises our software and hardware, allowing us to learn how we can continue to improve and adapt our Waymo Driver for the trucking domain.

Looking at Texas specifically: Dallas and Houston are known to be part of the biggest freight hubs in the US. Operating in that environment, we can test our Waymo Driver on highly dense highways and shipper lanes, further understand how other truck and passenger car drivers behave on these routes, and continue to refine the way our Waymo Driver reacts and responds in these busy driving regions. Additionally, it also enables us to test in a place with unique weather conditions that can help us drive our capabilities in that area forward.

Can you discuss the Waymo Open Dataset which includes both sensor data and labeled data, and the benefits to Waymo for sharing this valuable dataset?

At Waymo, we’re tackling some of the hardest problems that exist in machine learning. To aid the research community in making advancements in machine perception and self-driving technology, we’ve released the Waymo Open Dataset, which is one of the largest and most diverse publicly available fully self-driving datasets. Available at no cost to researchers at waymo.com/open, the dataset consists of 1,950 segments of high-resolution sensor data and covers a wide variety of environments, from dense urban centers to suburban landscapes, as well as data collected during day and night, at dawn and dusk, in sunshine and rain. In March 2020, we also launched the Waymo Open Dataset Challenges to provide the research community a way to test their expertise and see what others are doing.

In your personal opinion, how long will it be until the industry achieves true level 5 autonomy?

We have been working on this for over ten years now and so we have the benefit of that experience to know that this technology will come to the world step by step. Self-driving technology is so complex and we’ve gotten to where we are today because of advances in so many fields from sensing in hardware to machine learning. That’s why we’ve been taking a gradual approach to introduce this technology to the world. We believe it’s the safest and most responsible way to go, and we’ve also heard from our riders and partners that they appreciate this thoughtful and measured approach we’re taking to safely deploy this technology in their communities.

Thank you for the great interview, readers who wish to learn more should visit Waymo Via.