An ME and CSE team developed a more efficient machine learning method to train robotic controllers, deploying it on a two-legged robot that successfully navigated gravel, grass, hills, and more.
(text and background only visible when logged in)
Across sand, soggy grass, and gravel. Up slopes and stairs and across level ground. Whatever uneven terrain Georgia Tech researchers could find on campus or easily simulate presented no problem for their two-legged humanoid robot.
Machine learning Ph.D. student Feiyang Wu led development of a new kind of whole-body controller that allowed the humanoid robot to traverse all those varied surfaces. His method is computationally faster and cheaper than the leading approaches for training robotic controllers. And he was surprised how well it worked, even on surfaces that weren’t included in the training.
“For this bulky, very tall humanoid robot, it really hasn’t been proven that you can do agile locomotion on such austere terrain. Somehow our very efficient training recipe here can actually work for all kinds of terrain and environments,” said Wu, a machine learning Ph.D. student.
“We thought, OK, it seems reasonable to do locomotion on flat ground. But can the same [control] policy do rough terrain, especially in the real world? We were kind of skeptical, even though in simulation it looked not terrible, but not great.”
Wu presented the team’s training framework at the IEEE International Conference on Robotics and Automation, the world’s largest gathering of robotics researchers.
What makes the Georgia Tech controller work is a new take on training the robot’s control algorithms. Working with a team of researchers in the George W. Woodruff School of Mechanical Engineering and the School of Computational Science and Engineering (CSE), Wu reimagined a reinforcement learning technique that’s known as teacher and student learning.
This kind of machine learning method teaches a robotic controller how to behave through a simulated environment where a “teacher” agent is developed first. The teacher gets as much information as researchers can provide about the simulated environment, even complex data that couldn’t be realistically estimated or known in the real world. The teacher explores the simulation and learns how to move. Then it distills what’s it learned and teaches a new agent, a “student” robot, how to operate.
Wu said the method is analogous to a professor becoming an expert in a field. Then they teach students about that field — not necessarily a bunch of specific answers, but principles and concepts so the student can solve real-world problems. Similarly, the simulated teacher robot guides the student how to problem-solve navigating the real world using real robotic hardware.
(text and background only visible when logged in)
Top: The robot on a slippery floor surface using the manufacturer’s controller. Bottom: Using the Georgia Tech-trained controller, the robot is able to cross the same surface without difficulty.
“There are two problems with this approach,” Wu said. “It takes too much time to train them sequentially. Then, you’re wasting a lot of information that’s been gathered by the teacher.”
Training time is money when it comes to these simulations, because they require many hours of computation using expensive-to-use GPU chips.
The researchers’ solution? Train the teacher and the student at the same time.
“You don’t have to wait for the teacher to be an expert for it to begin teaching the student,” Wu said. “The teacher can gradually teach the student what they’ve learned along the way.”
To extend the professor-student metaphor, Wu said it’s like having a graduate student teach undergrads. The master’s or Ph.D. student is still learning themselves, but they have valuable knowledge that meets the needs of the undergrads.
Wu also adjusted the training so that the teacher robot learns from what the student robot experiences in the simulation. That helps close what robotics researchers call the teacher-student imitation gap — essentially that even with good instruction from the teacher, the student is still making decisions based on partial information. The student might encounter an obstacle or terrain and not have some important piece of data to properly navigate it.
“This gap can be very sinister and difficult to solve. We tried to mitigate it by letting the teacher learn from the data that’s been collected by the student as well,” Wu said. “The teacher experiences what’s possible for the student, and the hope is that helps the teacher offer better instruction.”
After applying their new approach and training a controller in simulations, they deployed it on a two-legged humanoid robot in Ye Zhao’s lab. It worked, allowing the robot to walk smoothly across a variety of surfaces.
The team also tried to forcefully push and pull the robot to see if they could disrupt its gait, but the robot adapted and adjusted to compensate.
Though Wu and his colleagues used a two-legged humanoid robot in their experiments, his “Learn to Teach” training framework is designed to be generic. It can be used for other robots with other configurations. It also can apply to other kinds of tasks besides walking.
Zhao, who co-advises Wu with CSE Assistant Professor Anqi Wu, said the control system performed better even than the controller provided by the robot’s manufacturer.
He said Wu brings a unique perspective to robotics work because of his background in machine learning.
“Most of the Ph.D. students in my lab come from a robotics and control background. Feiyang is starting to explore robotics problems from a more theoretical, algorithm-focused background, which is not easy,” said Zhao, associate professor and Woodruff Faculty Fellow in ME. “There’s a big barrier. If students get used to doing the programming and writing mathematics, they might not have the desire to explore working with the real hardware. Feiyang has a strong motivation to explore things on both sides, which is very unique.”
(text and background only visible when logged in)
About the Research
This research was supported by the Office of Naval Research, grant No. N000142312223; the U.S. Department of Agriculture, grant No. 2023-67021-41397; and the National Science Foundation, grant Nos. IIS-1924978, CMMI-2144309, and FRR-2328254. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any funding agency.
Preeminence in Research
Related Content
Humanoid Robots Make Strides Toward Stable Walking
ME researchers have created a first-of-its-kind strategy that gives two-legged robots a clear set of rules for reacting when something changes in its path.
Georgia Tech Research News
ME researchers have created a first-of-its-kind strategy that gives two-legged robots a clear set of rules for reacting when something changes in its path.
Smarter, Faster, and More Human: A Leap Toward General-Purpose Robots
A new AI system lets robots work faster than their human teachers without sacrificing accuracy.
Georgia Tech News Center
A new AI system lets robots work faster than their human teachers without sacrificing accuracy.
Real-World Helper Exoskeletons Just Got Closer to Reality
Georgia Tech researchers are using AI to quickly train exoskeleton devices, making it much more practical to develop, improve, and ultimately deploy wearable robots for people with impaired mobility.
Georgia Tech researchers are using AI to quickly train exoskeleton devices, making it much more practical to develop, improve, and ultimately deploy wearable robots for people with impaired mobility.