Princeton Robotics Seminar

In this talk, I will investigate the problem of learning manipulation skills across a diverse set of robotic embodiments. Conventionally, manipulation skills are learned separately for every task, environment and robot. However, in domains like Computer Vision and Natural Language Processing we have seen that one of the main contributing factors to generalisable models is large amounts of diverse data. If we were able to have one robot learn a new task even from data recorded with a different robot, then we could already scale up training data to a much larger degree for each robot embodiment. In this talk, I will present a new, large-scale datasets that was put together across multiple industry and academic research labs to make it possible to explore the possibility of cross-embodiment learning in the context of robotic manipulation, alongside experimental results that provide an example of effective cross-robot policies. Given this dataset, I will also present multiple alternative ways to learn cross-embodiment policies. These example approaches will include (1) UniGrasp - a model that allows to synthesise grasps with new hands, (2) XIRL - an approach to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos and (3) Equivact - an approach that leverages equivariance to learn sensorimotor policies that generalise to scenarios that are traditionally out-of-distribution.

Bio: Jeannette Bohg is an Assistant Professor of Computer Science at Stanford University. She was a group leader at the Autonomous Motion Department (AMD) of the MPI for Intelligent Systems until September 2017. Before joining AMD in January 2012, Jeannette Bohg was a PhD student at the Division of Robotics, Perception and Learning (RPL) at KTH in Stockholm. In her thesis, she proposed novel methods towards multi-modal scene understanding for robotic grasping. She also studied at Chalmers in Gothenburg and at the Technical University in Dresden where she received her Master in Art and Technology and her Diploma in Computer Science, respectively. Her research focuses on perception and learning for autonomous robotic manipulation and grasping. She is specifically interested in developing methods that are goal-directed, real-time and multi-modal such that they can provide meaningful feedback for execution and learning. Jeannette Bohg has received several Early Career and Best Paper awards, most notably the 2019 IEEE Robotics and Automation Society Early Career Award and the 2020 Robotics: Science and Systems Early Career Award.

---

PhD students and postdocs can signup to join Jeannette for lunch. There will also be a Robotics Social at 4:00 PM in the F-Wing Cafe Area - all are welcome to attend!

Safety Index is a special class of high order control barrier functions. Its purpose is to ensure forward invariance within a user-specified safe set and achieve finite time convergence to that set. Synthesizing a valid safety index poses significant challenges, particularly when dealing with control limits, uncertainties, and time-varying dynamics. In this talk, I will introduce a variety of approaches that can be used for safety index synthesis, including a rule-based method, an evolutionary optimization-based approach, a constrained reinforcement learning-based approach, an adversarial optimization-based approach, as well as sum of square programming. The parameterization of the safety index can either take an analytical form or be a neural network. I will conclude the talk by highlighting the limitations of existing work and discuss potential future directions, including integrating formal verification into neural safety index synthesis.

Bio: Dr. Changliu Liu is an assistant professor in the Robotics Institute, School of Computer Science, Carnegie Mellon University (CMU), where she leads the Intelligent Control Lab. Prior to joining CMU, Dr. Liu was a postdoc at Stanford Intelligent Systems Laboratory. She received her Ph.D. in Engineering together with Master degrees in Engineering and Mathematics from University of California at Berkeley and her bachelor degrees in Engineering and Economics from Tsinghua University. Her research interests lie in the design and verification of intelligent systems with applications to manufacturing and transportation. She published the book “Designing robot behavior in human-robot interactions” with CRC Press in 2019. She is the founder of the International Neural Network Verification Competition launched in 2020. Her work has been recognized by NSF Career Award, Amazon Research Award, Ford URP Award, Advanced Robotics for Manufacturing Champion Award, and many best/outstanding paper awards.

Students: Sign-up for lunch with the speaker here (12:00 - 1:30 PM.)

PhD students and postdocs can signup to join Ye for lunch here. There will also be a Robotics Social at 4:00 PM in the F-Wing Cafe Area - all are welcome to attend!

---

While legged robots have made remarkable progress in dynamic balancing and locomotion, there remains substantial room for improvement in terms of safe navigation and decision-making capabilities. One major challenge stems from the difficulty of designing safe, resilient, and real-time planning and decision-making frameworks for these complex legged machines navigating unstructured environments. Symbolic planning and distributed trajectory optimization offer promising yet underexplored solutions. This talk will introduce three perspectives on enhancing safety and resilience in task and motion planning (TAMP) for agile legged navigation. First, we'll discuss hierarchically integrated TAMP for dynamic locomotion in environments susceptible to perturbations, focusing on robust recovery behaviors. Next, we'll cover our recent work on safe and socially acceptable legged navigation planning in environments that are partially observable and crowded with humans. Lastly, we'll delve into distributed contact-aware trajectory optimization methods achieving dynamic consensus for agile locomotion behaviors.

Bio: Ye Zhao is an Assistant Professor at The George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology. He received his Ph.D. degree in Mechanical Engineering from The University of Texas at Austin in 2016. After that, he was a Postdoctoral Fellow at Agile Robotics Lab, Harvard University. At Georgia Tech, he leads the Laboratory for Intelligent Decision and Autonomous Robots. His research interest focuses on planning and decision-making algorithms of highly dynamic and contact-rich robots. He received the George W. Woodruff School Faculty Research Award at Georgia Tech in 2023, NSF CAREER Award in 2022, and ONR YIP Award in 2023. He serves as an Associate Editor of T-RO, TMECH, RA-L, and L-CSS. His co-authored work has received multiple paper awards, including the 2021 ICRA Best Automation Paper Award Finalist, the 2023 Best Paper Award at the NeurIPS Workshop on Touch Processing, and the 2016 IEEE-RAS Whole-Body Control Best Paper Award Finalist.

This talk will provide an overview of research activities at the ASCO lab, currently including robot-assisted surgical micro-manipulation and navigation of autonomous vehicles (aerial, underwater, or ground) under state and perception uncertainty. We will specifically focus on motion planning with built-in robustness guarantees, i.e. by aiming to certify expected performance before actual deployment. The core idea is to employ probably-approximately-correct (PAC) bounds on performance which are used as an objective function in control policy optimization. Such robust policies could then provide high-confidence performance guarantees, such as “with 99% chance the robot will reach its goal, while avoiding collisions with 99.9% chance”, and result in improved safety and reliability.

Bio: Marin Kobilarov is an Associate Professor at the Johns Hopkins University and a Principal Engineer at Zoox/Amazon. At JHU he leads the Autonomous Systems, Control and Optimization (ASCO) lab which develops algorithms and software for planning, learning, and control of autonomous robotic systems. Their focus is on computational theory at the intersection of planning and learning, and on the system integration and deployment of robots that can operate safely and efficiently in challenging environments.

---

PhD students & faculty: If you are interested in meeting with Marin on 3/8, please reach out to nsimon@princeton.edu.

PhD students: If you are interested in joining the speaker for lunch on 3/8 from 12:15 - 1:30 PM, please fill out this form.

Within the climate crisis, architecture and the built environment play an outsized role, both in contributions to current emissions and projections for the future.  Standard design and construction practices are often wasteful, expensive, and exploitative, undermining a mission to create spaces of dignity, comfort, and delight.  Within this context, this talk will present an alternative approach, focused on promoting design and construction methods that produce high-quality, low-carbon, and inexpensive outcomes, empowered by emerging advances in computational design and construction robotics.  In addition to general methods, the talk will present a range of recent projects, including robotic planning and assembly of complex yet highly efficient truss structures, robotics-enabled fabrication of low-cost, low-carbon earthen and concrete structures, and algorithmic design and fabrication approaches for unconventional and circular material use.

Bio: Caitlin Mueller is an Associate Professor at MIT's Department of Architecture and Department of Civil and Environmental Engineering, in the Building Technology Program, where she leads the Digital Structures research group.  She works at the creative interface of architecture, structural engineering, and computation, and focuses on new computational design and digital fabrication methods for innovative, high-performance buildings and structures that empower a more sustainable and equitable future. Mueller holds three degrees from MIT in Architecture, Computation, and Building Technology, and one from Stanford in Structural Engineering.  Her research is funded by federal agencies and industry partners, including the National Science Foundation, FEMA, the MIT Tata Center, the Dar Group, Holcim, Robert McNeel & Associates, and Altair Engineering.  Mueller has won best paper awards from the International Association of Shell and Spatial Structures, the Symposium on Geometry Processing, and the Journal of Mechanical Design, and was awarded the ACADIA Innovative Research Award of Excellence by the Association for Computer Aided Design in Architecture in 2021 and the Diversity Achievement Award from the Association of Collegiate Schools of Architecture in 2022.

In this talk we will discuss an emerging paradigm in differentiable reinforcement learning called “online nonstochastic control”. The new approach applies techniques from online convex optimization and convex relaxations to obtain new methods with provable guarantees for classical settings in optimal and robust control. Time permitting we will discuss recent extensions to nonlinear adaptive control and iterative planning, as well as model free reinforcement learning. 

This theory was, and continues to be, developed here in Princeton with numerous collaborators, including Naman Agarwal, Brian Bullins, Karan Singh, Max Simchowitz, Xinyi Chen, Ani Majumdar, Sham Kakade, Udaya Ghai, Edgar Minasyan, Paula Gradu, and many others.

Bio: Elad Hazan is a professor of computer science at Princeton University. His research focuses on the design and analysis of algorithms for basic problems in machine learning and optimization. Amongst his contributions are the co-invention of the AdaGrad algorithm for deep learning, and the first sublinear-time algorithms for convex optimization. He is the recipient of the Bell Labs prize, the IBM Goldberg best paper award twice, in 2012 and 2008, a European Research Council grant, a Marie Curie fellowship and twice the Google Research Award. He served on the steering committee of the Association for Computational Learning and has been program chair for COLT 2015. In 2017 he co-founded In8 inc. focusing on efficient optimization and control, acquired by Google in 2018. He is the co-founder and director of Google AI Princeton.

Reliable operation in everyday human environments – homes, offices, and businesses – remains elusive for today’s robotic systems.  A key challenge is diversity, as no two homes or businesses are exactly alike.  However, despite the innumerable unique aspects of any home, there are many commonalities as well, particularly about how objects are placed and used.  These commonalities can be captured in semantic representations, and then used to improve the autonomy of robotic systems by, for example, enabling robots to infer missing information in human instructions, efficiently search for objects, or manipulate objects more effectively.  In this talk, I will discuss recent advances in semantic reasoning, particularly focusing on semantics of everyday objects, household environments, and the development of robotic systems that intelligently interact with their world.

Bio: Sonia Chernova is an Associate Professor in the College of Computing at Georgia Tech.  She directs the Robot Autonomy and Interactive Learning lab, where her research focuses on the development of intelligent and interactive autonomous systems.  Chernova’s contributions span robotics and artificial intelligence, including semantic reasoning, adaptive autonomy, human-robot interaction, and explainable AI.  She also leads the NSF AI Institute for Collaborative Assistance and Responsive Interaction for Networked Groups (AI-CARING), whose mission is to develop collaborative AI partners-in-care that help support a growing population of older adults, helping them sustain independence, improve quality of life, and increase effectiveness of care coordination across their care network.

Multi-robot systems are becoming more pervasive all around us, in the form of fleets of autonomous vehicles, future delivery drones, and robotic teammates for search and rescue.  As a result, it becomes increasingly critical to question the robustness of their coordination algorithms to reliable information exchange, security threats and/or corrupted data. This talk will focus on the role of control and information exchange for enhancing situational awareness and security of multirobot systems. An example is the consensus problem where classical results hold that agreement cannot be reached when malicious agents make up more than half of the network connectivity; this quickly leads to limitations in the practicality of many multi-robot coordination tasks. However, with the growing prevalence of cyber-physical systems comes novel opportunities for detecting attacks by using cross-validation with physical channels of information. In this talk we consider the class of problems where the probability of a particular (i,j) link being trustworthy is available as a random variable. We refer to these as “stochastic observations of trust.” We show that under this model, strong performance guarantees such as convergence for the consensus problem can be recovered, even in the case where the number of malicious agents is greater than ½ of the network connectivity and consensus would otherwise fail. We will present both a theoretical framework, and experimental results, for provably securing multi-robot distributed algorithms through careful use of communication.  Lastly, we will present promising results on new communication-centric methods for learning and sequential decision-making in tomorrow’s multi-robot systems.

Bio: Stephanie is an Assistant Professor in the John A. Paulson School of Engineering and Applied Sciences (SEAS) at Harvard University. Her work centers around trust and coordination in multi-robot systems for which she has received the Office of Naval Research Young Investigator award (2021) and the National Science Foundation CAREER award (2019). She has also been selected as a 2020 Sloan Research Fellow for her contributions at the intersection of robotics and communication. She has held a Visiting Assistant Professor position at Stanford University during the summer of 2019, and an Assistant Professorship at Arizona State University from 2018-2020. She completed her Ph.D. work (2014) on multi-robot coordination and control and her M.S. work (2009) on system identification and model learning. At MIT she collaborated extensively with the wireless communications group NetMIT, the result of which were two U.S. patents recently awarded in adaptive heterogeneous networks for multi-robot systems and accurate indoor positioning using Wi-Fi.  She completed her B.S. at Cornell University.

At the Toyota Research Institute (TRI), we've been working on behavior cloning for dexterous manipulation. Building on the Diffusion Policy framework that we've recently developed in collaboration with Shuran Song, we now have a very solid pipeline for taking ~50-100 bimanual haptic teleop demonstrations and turning that into a surprisingly effective visuomotor (+tactile) policy. Because there is no explicit state representation required, these skills work equally well manipulating deformable, liquid, or other difficult to model tasks as they do for more traditional rigid-object manipulation. We're actively scaling this up into the multi-task setting and now see a plausible path towards "Large Behavior Models."

Bio: Russ Tedrake is the Toyota Professor at the Massachusetts Institute of Technology (MIT) in the Department of Electrical Engineering and Computer Science, Mechanical Engineering, and Aero/Astro, and he is a member of MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). He is also the Vice President of Robotics Research at Toyota Research Institute (TRI). He received a B.S.E. in Computer Engineering from the University of Michigan in 1999, and a Ph.D. in Electrical Engineering and Computer Science from MIT in 2004. Dr. Tedrake is the Director of the MIT CSAIL Center for Robotics and was the leader of MIT’s entry in the DARPA Robotics Challenge. He is a recipient of the NSF CAREER Award, the MIT Jerome Saltzer Award for undergraduate teaching, the DARPA Young Faculty Award in Mathematics, the 2012 Ruth and Joel Spira Teaching Award, and was named a Microsoft Research New Faculty Fellow. His research has been recognized with numerous conference best paper awards, including ICRA, Robotics: Science and Systems, Humanoids, Hybrid Systems: Computation and Control, as well as the inaugural best paper award from the IEEE RAS Technical Committee on Whole-Body Control.

Robots hold the promise of serving human needs, like helping older adults live independently at home or assisting drivers in preventing crashes. For these robots to integrate seamlessly into people's lives, they must provide proactive assistance that is responsive to their human partners' needs. Often, these needs are a result of underlying mental states like intent or awareness. Conversely, it is also useful for people to have an accurate mental model of their robot assistant's policy and knowledge. Mental states may be revealed implicitly through actions the agents take, such as gazing at a certain object or moving in a certain way. This talk describes research on developing collaborative robots that infer people's needs through interaction, adapt to people's individual preferences, and communicate their own models to make the interaction more explainable. These robots are evaluated in a range of human-robot interaction domains, such as manipulation and driving.

Bio: Dr. Henny Admoni is an Associate Professor in the Robotics Institute at Carnegie Mellon University, where she leads the Human And Robot Partners (HARP) Lab. Dr. Admoni’s research interests include human-robot interaction, assistive robotics, and nonverbal communication. Dr. Admoni holds a PhD in Computer Science from Yale University, and a BA/MA joint degree in Computer Science from Wesleyan University.