Teaching

Robot learning has made significant strides in just the last 2 years. For the first time, there exist “generalist” manipulation systems that one can download from the web, and deploy on their robots and environments without substantial finetuning, and achieve non-trivial task success rates. At the same time, making robot learning models (a.k.a. “policies”) work, and generalize across a fairly broad set of tasks and environments, is an open research problem. This has caused an undeniable shift in the skillsets that robotics practitioners (engineers, scientists) must now have, to get into (and succeed at) frontier robotics labs.

This course aims to address that gap. In this class, we will take a “full-stack” perspective of robot learning – we will understand (and build) robot hardware (i.e., a small tabletop robot arm) and discuss the entirety of the modern robot learning lifecycle. This includes task design and selection, curating and processing data, training neural network models on the processed data, deploying these trained models, and evaluating how the trained approaches fare.

This is not an introductory class. We expect students to already have requisite knowledge in deep learning (training and debugging transformer models, diffusion models) or be able to pick them up independently. We will do an extremely short recap of the essentials for the class (one 2.5-hour lecture). We then will look at various data curation and processing strategies, single-task imitation learning, multi-task policy learning, and will discuss some of the frontiers (world models, for example).

This is a collaborative class. Assignments will be completed in teams of about 3 students each. Each team will receive an SO-101 robot arm kit and will assemble their robot from scratch. There are no written exams.

Roughly half of your time in this course will be spent outside the classroom, building, collecting data on, deploying policies on, and debugging, the SO-101 robot arm. Robot hardware work is hard; that is a large part of what this course is about. Brace for hardware and/or software failures happening frequently, and inevitably before/during deadlines/demos.