Differentiable Particles for Object Manipulation

This study introduces novel methods for particle-based object manipulation, focusing on unknown rigid and deformable objects. Using the differentiable particle representation, our methods can reason about the dynamics of deformable objects, supporting better robot manipulation.

Differentiable Algorithm Network (DAN)

How do we exploit both prior knowledge—geometry, physics, logic, etc.—and data-driven deep learning for robust decision making?  The key idea of DAN is a unified neural policy representation, encoding both a learned system model and an algorithm that solves the model. The policy network is fully differentiable and trained end-to-end from data.

Intention-Net for Autonomous Navigation

Humans can navigate capably in the rich 3D world, using only a schematic 2D floor map. iNet is a novel navigation architecture aiming to give robots the same capability.

Integrating Planning and Learning for Large-Scale Decision Making under Uncertainty

Real-time planning under uncertainty is critical for robots operating in complex, fast-changing environments, e.g.,  an autonomous robot vehicle driving in dense traffic of cars, bicycles, pedestrians … Our new algorithm LeTS-Drive tackles the challenge  by integrating planning and learning in a closed loop so that the planner and the learner inform each other and  improve in synchrony. The algorithm learns on its own in a self-supervised manner, without human effort on data labeling.

Natural Language Interaction with Robots

How would you want to use robots in the future? Speak the instructions to them! Robots must then connect human language instructions, visual perception of the physical world, and actions to accomplish the task. This project addresses several key questions, including visual grounding and interactive disambiguation.

Object Manipulation under Uncertainty

How can service robots manipulate a wide variety of everyday objects? One main challenge is uncertainty in object shape, pose, and physical properties. We investigate several planning and learning methods for robust object manipulation when object shape, physical properties, … are unknown in advance.

Autonomous Driving in Traffic: Modeling, Simulation, and Planning

Accurate and efficient motion prediction is critical for autonomous driving systems.  We propose general motion prediction models that enables large-scale real-time simulation and planning for heterogeneous, interactive traffic agents, including cars, buses, bicycles, pedestrians, …

Reinforcement Learning with Complex Observations

While reinforcement learning has achieved great success in simulated environments, such as Atari games, Go, and even Dota,  realistic physical environments post new challenges with complex partial observations. Key to our approach is to learn from partial observations a robust and compact latent state representation.

Trust Models for Human-Robot Collaboration

Can we humans trust robots? Can robots act in a way to win our trust? We aim to develop a formal computational framework of trust to support automated decision making for fluid collaboration between humans and robots.

Reinventing User Interaction with Flying Cameras

In the future, many people may take along compact flying cameras—a camera mounted on a flying drone—and use their touchscreen mobile devices as viewfinders to take photos. This project explores the user interaction design and system implementation issues  for a flying camera, which  leverages the autonomous flying capability of a drone-mounted camera for a great photo-taking experience.

Online POMDP Planning

Robust decision making under uncertainty requires us to hedge against myriad future possibilities, resulting in computational intractability. Maybe, it is sufficient to consider just a  subset of representative samples. This is the key idea behind DESPOT, a state-of-the-art online search algorithm for approximate POMDP planning in dynamic environments.

This project addresses the key challenges in building fast algorithms and software for solving partially observable Markov decision processes (POMDPs). APPL is a software package that implements our POMDP solvers.

We want to build autonomous robots that can follow people and recognize their activities. Such capabilities are important in applications such as home care for the elderly, smart environments, and iteractive media.

We aim to develop reliable and efficient motion strategies for an autonomous robot to follow a target and keep it within sensing range.  We investigate several greedy strategies using only local sensor information to achieve the goal.

Several advanced sampling strategies have been proposed in recent years to address the narrow passage problem for probabilistic roadmap (PRM) planning.  All have unique strengths, but none of them solves the problem completely. We investigate general and systematic approaches for adaptively combining multiple sampling strategies so that their individual strengths are preserved.

Motion planning is concerned with computing collision-free motion for moving objects (e.g., drug molecules inside the binding cavity of a protein, robot manipulators on an assembly line, or articulated models of human arms) in an environment populated with obstacles. The key difficulty of motion planning is its exponential dependence on the number of degrees of freedom of a moving object. The focus of this work is to use random sampling as a fundamental tool for attacking this challenge.

A car cannot move sidewise. A bouncing ball must obey the laws of physics. These kinematic or dynamic constraints compound the difficulty of motion planning. We have extended our sampling techniques for path planning to handle objects whose motion is described by a control system, which is a set of differential equations that captures diverse types of kinematic and dynamic constraints. The implemented algorithm is fast enough to deal with moving obstacles in “real-time”.

Proper placement of the base of a robot manipulator is an important issue in many robotics applications. For instance, in manufacturing, the base location of a manipulator has a significant impact on the cycle time of tasks such as spot welding and inspection. This work uses randomized motion planning as the main tool to search for the near-optimal placement.